2023-12-03T11:31:25.318 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-03T11:31:25.319 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150 --owner scheduled_alimasa@teuthology --description rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} --name alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi -- /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/orig.config.yaml 2023-12-03T11:31:25.365 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T11:31:25.448 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150 branch: wip-alimasa-multi-pubsub-etag description: rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} email: null first_in_suite: false job_id: '7476150' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-alimasa-multi-pubsub-etag ceph: conf: client: debug rgw: 20 rgw bucket counters cache: true rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw max chunk size: 4194304 rgw obj stripe size: 6291456 rgw torrent flag: true rgw user counters cache: true setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 lockdep: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled - \(OSD_SLOW_PING_TIME sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 openssl_keys: rgw.client.0: ca: root client: client.0 embed-key: true root: client: client.0 cn: teuthology install: - client.0 key-type: rsa:4096 rgw: client.0: ssl certificate: rgw.client.0 compression type: random ec-data-pool: true frontend: beast inline data: false storage classes: LUKEWARM, FROZEN s3tests: force-branch: ceph-master slow_backend: true storage classes: LUKEWARM, FROZEN thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-alimasa-multi-pubsub-etag sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 owner: scheduled_alimasa@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 sleep_before_teardown: 0 suite: rgw suite_branch: wip-alimasa-multi-pubsub-etag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWklY4I2diZT214tI3xTz6GRNhwlcyPni6UlOvuW/o+2IoxhJOzik5ptxGmaxuz2N7CpKMa/Odz5B5mVb3znSg= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsJ+OYRQMdJGRs6a9/kmeo3WSNgQsUqBVDEgU4jFR4A5K8Xqv6bW0WMEOXZjF3goVFqedsSiXKs+J2O59oiA7w= tasks: - install: extra_system_packages: deb: - s3cmd rpm: - s3cmd - ceph: null - openssl_keys: null - rgw: client.0: null - tox: - client.0 - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_rgw_gc.sh - cls/test_cls_rgw_stats.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_librgw_file.sh - workunit: clients: client.0: - rgw/test_rgw_s3_mp_reupload.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-03_11:04:43 tube: smithi user: alimasa verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-03T11:31:25.448 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa; will attempt to use it 2023-12-03T11:31:25.449 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks 2023-12-03T11:31:25.449 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-03T11:31:25.450 INFO:teuthology.task.internal:Checking packages... 2023-12-03T11:31:25.474 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bcbf278cf395873c2a482f5a7b2345faf2160319' 2023-12-03T11:31:25.474 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-03T11:31:25.474 INFO:teuthology.packaging:ref: None 2023-12-03T11:31:25.474 INFO:teuthology.packaging:tag: None 2023-12-03T11:31:25.474 INFO:teuthology.packaging:branch: wip-alimasa-multi-pubsub-etag 2023-12-03T11:31:25.474 INFO:teuthology.packaging:sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:31:25.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-alimasa-multi-pubsub-etag 2023-12-03T11:31:25.618 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7537.gbcbf278c 2023-12-03T11:31:25.618 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-03T11:31:25.626 INFO:teuthology.task.internal:no buildpackages task found 2023-12-03T11:31:25.626 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-03T11:31:25.632 INFO:teuthology.task.internal:Saving configuration 2023-12-03T11:31:25.648 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-03T11:31:25.654 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T11:31:25.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:26:17.977366', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWklY4I2diZT214tI3xTz6GRNhwlcyPni6UlOvuW/o+2IoxhJOzik5ptxGmaxuz2N7CpKMa/Odz5B5mVb3znSg='} 2023-12-03T11:31:25.703 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:26:17.975308', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsJ+OYRQMdJGRs6a9/kmeo3WSNgQsUqBVDEgU4jFR4A5K8Xqv6bW0WMEOXZjF3goVFqedsSiXKs+J2O59oiA7w='} 2023-12-03T11:31:25.703 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-03T11:31:25.709 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-03T11:31:25.709 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-03T11:31:25.709 INFO:teuthology.run_tasks:Running task console_log... 2023-12-03T11:31:25.783 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa556200f70>, signals=[15]) 2023-12-03T11:31:25.783 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-03T11:31:25.790 INFO:teuthology.task.internal:Opening connections... 2023-12-03T11:31:25.790 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:31:25.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:31:25.881 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:31:25.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:31:25.966 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-03T11:31:25.974 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-12-03T11:31:25.991 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-12-03T11:31:25.992 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-12-03T11:31:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2023-12-03T11:31:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2023-12-03T11:31:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2023-12-03T11:31:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2023-12-03T11:31:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T11:31:26.050 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T11:31:26.051 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-12-03T11:31:26.069 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2023-12-03T11:31:26.086 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2023-12-03T11:31:26.087 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2023-12-03T11:31:26.143 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T11:31:26.144 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T11:31:26.144 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-12-03T11:31:26.160 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-03T11:31:26.170 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-03T11:31:26.189 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-03T11:31:26.189 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-12-03T11:31:26.192 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2023-12-03T11:31:26.207 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-03T11:31:26.213 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-03T11:31:26.213 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-12-03T11:31:26.249 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2023-12-03T11:31:26.274 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-03T11:31:26.330 INFO:teuthology.run_tasks:Running task kernel... 2023-12-03T11:31:26.347 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-03T11:31:26.347 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-03T11:31:26.347 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T11:31:26.348 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T11:31:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:31:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:31:26.364 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-12-03T11:31:26.364 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2023-12-03T11:31:26.420 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2023-12-03T11:31:26.420 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2023-12-03T11:31:26.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2023-12-03T11:31:26.421 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2023-12-03T11:31:26.422 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2023-12-03T11:31:26.422 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2023-12-03T11:31:28.693 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.4 MB 00:00 2023-12-03T11:31:28.750 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.4 MB 00:00 2023-12-03T11:31:30.367 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-12-03T11:31:30.427 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2023-12-03T11:31:31.176 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-12-03T11:31:31.825 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 7.9 MB 00:01 2023-12-03T11:31:34.734 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-12-03T11:31:38.643 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-03T11:31:39.726 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 18 MB 00:07 2023-12-03T11:31:40.285 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2023-12-03T11:31:42.005 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-12-03T11:31:44.585 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.8 MB 00:01 2023-12-03T11:31:46.086 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2023-12-03T11:31:46.778 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-03T11:31:47.288 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2023-12-03T11:31:49.126 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T11:31:49.127 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T11:31:49.238 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T11:31:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 96 M 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 121 M 2023-12-03T11:31:49.251 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-12-03T11:31:50.054 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 10 MB/s | 5.6 MB 00:00 2023-12-03T11:31:50.371 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 24 MB/s | 20 MB 00:00 2023-12-03T11:31:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 20 MB/s | 38 MB 00:01 2023-12-03T11:31:51.722 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 19 MB/s | 32 MB 00:01 2023-12-03T11:31:51.722 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:31:51.722 INFO:teuthology.orchestra.run.smithi116.stdout:Total 39 MB/s | 96 MB 00:02 2023-12-03T11:31:52.196 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-12-03T11:31:52.578 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-12-03T11:31:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-12-03T11:31:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-12-03T11:31:54.234 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-12-03T11:31:54.235 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-12-03T11:31:54.561 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T11:31:54.562 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T11:31:54.660 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-03T11:31:54.670 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-03T11:31:54.671 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2023-12-03T11:31:54.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:31:54.672 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 96 M 2023-12-03T11:31:54.672 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 121 M 2023-12-03T11:31:54.672 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-03T11:31:55.306 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 9.4 MB/s | 5.6 MB 00:00 2023-12-03T11:31:55.981 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-modules-core-5.14.0-388.el9.x86_6 48 MB/s | 32 MB 00:00 2023-12-03T11:31:56.673 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 19 MB/s | 38 MB 00:01 2023-12-03T11:31:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-12-03T11:31:57.032 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-core-5.14.0-388.el9.x86_64.rpm 8.8 MB/s | 20 MB 00:02 2023-12-03T11:31:57.032 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:31:57.032 INFO:teuthology.orchestra.run.smithi152.stdout:Total 41 MB/s | 96 MB 00:02 2023-12-03T11:31:57.511 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:31:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:31:57.883 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-03T11:31:58.260 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-03T11:31:58.260 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-03T11:31:59.570 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-03T11:31:59.571 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-03T11:32:00.396 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:32:00.732 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:02.488 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-03T11:32:03.125 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:32:03.252 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:32:05.321 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:05.523 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:06.056 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:32:06.224 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:10.821 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:11.023 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:15.641 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:38.835 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:38.877 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:39.266 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:39.267 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:32:39.267 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:32:39.267 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:39.682 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:32:39.683 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-12-03T11:32:40.140 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:32:40.303 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:40.304 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:32:40.304 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T11:32:40.304 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T11:32:40.304 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T11:32:40.304 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-03T11:32:40.304 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T11:32:40.305 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:32:40.421 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:40.422 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:32:40.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T11:32:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:32:40.457 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T11:32:40.458 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T11:32:40.458 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-388.el9.x86_64 2023-12-03T11:32:40.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T11:32:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-12-03T11:32:42.413 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T11:32:42.449 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-12-03T11:32:42.451 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-12-03T11:32:42.466 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.hn8aPDc1Ly 2023-12-03T11:32:42.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hn8aPDc1Ly 2023-12-03T11:32:42.535 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.hn8aPDc1Ly 2023-12-03T11:32:42.678 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.hn8aPDc1Ly is 6KB 2023-12-03T11:32:42.727 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.hn8aPDc1Ly 2023-12-03T11:32:42.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T11:32:42.806 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T11:32:42.806 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T11:32:42.806 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T11:32:42.806 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T11:32:42.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T11:32:42.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-12-03T11:32:44.437 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:44.480 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:44.909 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:44.909 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:32:44.909 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:32:44.909 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:32:45.340 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:32:45.341 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-03T11:32:45.867 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:32:46.050 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:46.051 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:32:46.051 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T11:32:46.051 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T11:32:46.051 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T11:32:46.051 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-03T11:32:46.052 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T11:32:46.052 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:32:46.170 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:32:46.170 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:32:46.170 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2023-12-03T11:32:46.214 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2023-12-03T11:32:46.214 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2023-12-03T11:32:46.215 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2023-12-03T11:32:46.216 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2023-12-03T11:32:46.217 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:32:46.218 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T11:32:46.219 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T11:32:46.219 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-388.el9.x86_64 2023-12-03T11:32:46.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T11:32:46.901 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2023-12-03T11:32:48.412 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T11:32:48.443 INFO:teuthology.orchestra.run.smithi152.stderr:done 2023-12-03T11:32:48.446 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2023-12-03T11:32:48.461 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.mI6zI42EXB 2023-12-03T11:32:48.462 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mI6zI42EXB 2023-12-03T11:32:48.529 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.mI6zI42EXB 2023-12-03T11:32:48.663 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.mI6zI42EXB is 6KB 2023-12-03T11:32:48.712 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.mI6zI42EXB 2023-12-03T11:32:48.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T11:32:48.790 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T11:32:48.790 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T11:32:48.790 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T11:32:48.791 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T11:32:48.792 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T11:32:48.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2023-12-03T11:33:12.970 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-03T11:33:12.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:33:12.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:33:18.940 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-03T11:33:18.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:33:18.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:33:31.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-12-03T11:33:40.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:33:40.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:33:43.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-12-03T11:33:53.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-12-03T11:33:55.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:33:55.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:34:02.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:34:02.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:34:13.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-12-03T11:34:21.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-12-03T11:34:28.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:34:28.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:34:29.162 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-12-03T11:34:29.696 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:34:29.997 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-03T11:34:29.998 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-12-03T11:34:30.021 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-388.el9.x86_64 2023-12-03T11:34:30.021 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T11:34:30.022 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T11:34:30.022 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T11:34:31.023 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-03T11:34:31.023 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T11:34:31.147 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-12-03T11:34:31.401 DEBUG:teuthology.parallel:result is None 2023-12-03T11:34:33.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:34:33.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:34:36.523 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-12-03T11:34:37.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:34:37.025 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-388.el9.x86_64"... 2023-12-03T11:34:37.025 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2023-12-03T11:34:37.038 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-388.el9.x86_64 2023-12-03T11:34:37.039 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T11:34:37.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T11:34:37.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T11:34:38.039 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-03T11:34:38.039 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T11:34:38.140 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2023-12-03T11:34:38.190 DEBUG:teuthology.parallel:result is None 2023-12-03T11:34:38.190 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-03T11:34:38.198 INFO:teuthology.task.internal:Creating test directory... 2023-12-03T11:34:38.198 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-03T11:34:38.201 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-03T11:34:38.216 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-03T11:34:38.222 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-03T11:34:38.233 INFO:teuthology.task.internal:Creating archive directory... 2023-12-03T11:34:38.233 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-03T11:34:38.258 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-03T11:34:38.302 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-03T11:34:38.308 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-03T11:34:38.309 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-03T11:34:38.327 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-03T11:34:38.361 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:34:38.362 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:34:38.375 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:34:38.375 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:34:38.377 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-03T11:34:38.383 INFO:teuthology.task.internal:Configuring sudo... 2023-12-03T11:34:38.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-03T11:34:38.419 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-03T11:34:38.447 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-03T11:34:38.456 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-03T11:34:38.456 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-03T11:34:38.489 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-03T11:34:38.503 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:34:38.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:34:38.634 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:34:38.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:34:38.759 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:34:38.760 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-03T11:34:38.826 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:34:38.868 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:34:38.916 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:34:38.999 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:34:39.039 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:34:39.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-03T11:34:39.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-12-03T11:34:39.114 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2023-12-03T11:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T11:34:39.208 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T11:34:39.666 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-03T11:34:39.675 INFO:teuthology.task.internal:Starting timer... 2023-12-03T11:34:39.675 INFO:teuthology.run_tasks:Running task pcp... 2023-12-03T11:34:39.685 INFO:teuthology.run_tasks:Running task selinux... 2023-12-03T11:34:39.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2023-12-03T11:34:39.759 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2023-12-03T11:34:39.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2023-12-03T11:34:39.815 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2023-12-03T11:34:39.817 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-03T11:34:39.817 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2023-12-03T11:34:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2023-12-03T11:34:39.844 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2023-12-03T11:34:39.859 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2023-12-03T11:34:39.859 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2023-12-03T11:34:39.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T11:34:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:34:39.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T11:34:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:34:39.942 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-03T11:34:39.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2023-12-03T11:34:39.987 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2023-12-03T11:34:40.022 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-03T11:34:40.032 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-03T11:34:40.037 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-03T11:34:40.163 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-03T11:34:40.178 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-03T11:34:40.179 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-03T11:40:05.917 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2023-12-03T11:40:05.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:40:05.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:40:06.005 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-12-03T11:40:06.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-12-03T11:40:06.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:40:06.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:40:06.161 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-12-03T11:40:06.229 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-03T11:40:06.230 INFO:teuthology.run_tasks:Running task clock... 2023-12-03T11:40:06.241 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-03T11:40:06.241 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-03T11:40:06.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:40:06.244 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-03T11:40:06.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:40:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-03T11:40:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-03T11:40:06.300 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-03T11:40:06.317 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-03T11:40:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2023-12-03T11:40:06.331 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2023-12-03T11:40:06.347 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2023-12-03T11:40:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-03T11:40:06.360 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2023-12-03T11:40:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-03T11:40:06.377 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-03T11:40:06.393 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-03T11:40:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.410 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.439 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:40:06.440 INFO:teuthology.run_tasks:Running task install... 2023-12-03T11:40:06.449 DEBUG:teuthology.task.install:project ceph 2023-12-03T11:40:06.449 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}} 2023-12-03T11:40:06.449 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'} 2023-12-03T11:40:06.449 INFO:teuthology.task.install:Using flavor: default 2023-12-03T11:40:06.455 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-03T11:40:06.455 INFO:teuthology.task.install:extra packages: [] 2023-12-03T11:40:06.455 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'tag': None, 'wait_for_package': False} 2023-12-03T11:40:06.456 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:40:06.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'tag': None, 'wait_for_package': False} 2023-12-03T11:40:06.458 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:40:06.612 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/ 2023-12-03T11:40:06.613 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7537.gbcbf278c 2023-12-03T11:40:06.682 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/ 2023-12-03T11:40:06.682 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7537.gbcbf278c 2023-12-03T11:40:06.804 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-03T11:40:06.804 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:40:06.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-03T11:40:06.807 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-03T11:40:06.807 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:40:06.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-03T11:40:06.832 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, s3cmd, s3cmd, s3cmd on remote rpm x86_64 2023-12-03T11:40:06.832 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bcbf278cf395873c2a482f5a7b2345faf2160319/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-03T11:40:06.837 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, s3cmd, s3cmd, s3cmd, s3cmd on remote rpm x86_64 2023-12-03T11:40:06.837 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bcbf278cf395873c2a482f5a7b2345faf2160319/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-03T11:40:06.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-03T11:40:06.905 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-03T11:40:06.980 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-03T11:40:06.987 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-03T11:40:07.046 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2023-12-03T11:40:07.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2023-12-03T11:40:07.053 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2023-12-03T11:40:07.054 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2023-12-03T11:40:07.304 INFO:teuthology.orchestra.run.smithi152.stdout:73 files removed 2023-12-03T11:40:07.305 INFO:teuthology.orchestra.run.smithi116.stdout:73 files removed 2023-12-03T11:40:07.331 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd s3cmd s3cmd s3cmd s3cmd 2023-12-03T11:40:07.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd s3cmd s3cmd s3cmd s3cmd 2023-12-03T11:40:07.986 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 230 kB/s | 78 kB 00:00 2023-12-03T11:40:08.038 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 203 kB/s | 78 kB 00:00 2023-12-03T11:40:08.403 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 26 kB/s | 9.5 kB 00:00 2023-12-03T11:40:08.464 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-03T11:40:08.770 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2023-12-03T11:40:08.873 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2023-12-03T11:40:09.164 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-03T11:40:09.409 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-03T11:40:10.523 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-12-03T11:40:10.751 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-12-03T11:40:11.533 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 7.9 MB 00:00 2023-12-03T11:40:11.555 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-12-03T11:40:13.935 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 15 MB/s | 18 MB 00:01 2023-12-03T11:40:14.398 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-12-03T11:40:18.102 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 5.8 MB 00:00 2023-12-03T11:40:19.160 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 5.8 MB 00:01 2023-12-03T11:40:19.395 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 15 kB 00:00 2023-12-03T11:40:20.766 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-03T11:40:21.815 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 20 MB 00:02 2023-12-03T11:40:24.359 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 20 MB 00:03 2023-12-03T11:40:26.538 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-12-03T11:40:29.100 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-12-03T11:40:29.493 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:40:29.494 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:40:29.563 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 6.6 k 2023-12-03T11:40:29.582 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 5.2 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 860 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 141 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 1.5 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 133 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 1.8 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 7.4 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 50 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 8.6 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 41 M 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 262 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 774 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 31 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 718 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 126 k 2023-12-03T11:40:29.583 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 162 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 320 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 298 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 99 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 85 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 170 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: s3cmd noarch 2.3.0-4.el9 epel 197 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.3 M 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-03T11:40:29.584 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 19 M 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 28 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 2.1 M 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 243 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.4 M 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 18 M 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 15 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 24 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-03T11:40:29.585 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 164 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 475 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.8 M 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-03T11:40:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 45 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 129 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-03T11:40:29.587 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-03T11:40:29.588 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-03T11:40:29.589 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-03T11:40:29.590 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout:Install 118 Packages 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2023-12-03T11:40:29.591 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:40:29.592 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 181 M 2023-12-03T11:40:29.592 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-03T11:40:30.383 INFO:teuthology.orchestra.run.smithi152.stdout:(1/120): ceph-18.0.0-7537.gbcbf278c.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2023-12-03T11:40:30.691 INFO:teuthology.orchestra.run.smithi152.stdout:(2/120): ceph-base-18.0.0-7537.gbcbf278c.el9.x8 9.7 MB/s | 5.2 MB 00:00 2023-12-03T11:40:30.749 INFO:teuthology.orchestra.run.smithi152.stdout:(3/120): ceph-immutable-object-cache-18.0.0-753 2.4 MB/s | 141 kB 00:00 2023-12-03T11:40:30.840 INFO:teuthology.orchestra.run.smithi152.stdout:(4/120): ceph-fuse-18.0.0-7537.gbcbf278c.el9.x8 1.8 MB/s | 860 kB 00:00 2023-12-03T11:40:30.924 INFO:teuthology.orchestra.run.smithi152.stdout:(5/120): ceph-mds-18.0.0-7537.gbcbf278c.el9.x86 12 MB/s | 2.1 MB 00:00 2023-12-03T11:40:31.258 INFO:teuthology.orchestra.run.smithi152.stdout:(6/120): ceph-mon-18.0.0-7537.gbcbf278c.el9.x86 13 MB/s | 4.4 MB 00:00 2023-12-03T11:40:31.375 INFO:teuthology.orchestra.run.smithi152.stdout:(7/120): ceph-mgr-18.0.0-7537.gbcbf278c.el9.x86 2.7 MB/s | 1.5 MB 00:00 2023-12-03T11:40:31.780 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:40:31.780 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:40:31.850 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-12-03T11:40:31.864 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-12-03T11:40:31.864 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 6.6 k 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 5.2 M 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 860 k 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 141 k 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 1.5 M 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 133 k 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 1.8 M 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 7.4 M 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 50 k 2023-12-03T11:40:31.865 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 8.6 M 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 41 M 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 262 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 774 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 31 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 718 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 126 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 162 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 320 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 298 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 99 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 85 k 2023-12-03T11:40:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 170 k 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: s3cmd noarch 2.3.0-4.el9 epel 197 k 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.3 M 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 19 M 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 28 k 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 2.1 M 2023-12-03T11:40:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 243 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.4 M 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 18 M 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 15 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 24 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-03T11:40:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 164 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 475 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.8 M 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-03T11:40:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 45 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 129 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-03T11:40:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-03T11:40:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-03T11:40:31.872 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-03T11:40:31.873 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Install 118 Packages 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2023-12-03T11:40:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:40:31.875 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 181 M 2023-12-03T11:40:31.875 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-12-03T11:40:32.435 INFO:teuthology.orchestra.run.smithi116.stdout:(1/120): ceph-18.0.0-7537.gbcbf278c.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2023-12-03T11:40:32.583 INFO:teuthology.orchestra.run.smithi152.stdout:(8/120): ceph-osd-18.0.0-7537.gbcbf278c.el9.x86 13 MB/s | 18 MB 00:01 2023-12-03T11:40:32.719 INFO:teuthology.orchestra.run.smithi116.stdout:(2/120): ceph-fuse-18.0.0-7537.gbcbf278c.el9.x8 3.0 MB/s | 860 kB 00:00 2023-12-03T11:40:32.810 INFO:teuthology.orchestra.run.smithi116.stdout:(3/120): ceph-immutable-object-cache-18.0.0-753 1.5 MB/s | 141 kB 00:00 2023-12-03T11:40:32.842 INFO:teuthology.orchestra.run.smithi152.stdout:(9/120): ceph-common-18.0.0-7537.gbcbf278c.el9. 7.1 MB/s | 19 MB 00:02 2023-12-03T11:40:32.859 INFO:teuthology.orchestra.run.smithi152.stdout:(10/120): ceph-selinux-18.0.0-7537.gbcbf278c.el 87 kB/s | 24 kB 00:00 2023-12-03T11:40:32.934 INFO:teuthology.orchestra.run.smithi152.stdout:(11/120): libcephfs-devel-18.0.0-7537.gbcbf278c 415 kB/s | 31 kB 00:00 2023-12-03T11:40:32.969 INFO:teuthology.orchestra.run.smithi116.stdout:(4/120): ceph-base-18.0.0-7537.gbcbf278c.el9.x8 6.5 MB/s | 5.2 MB 00:00 2023-12-03T11:40:33.386 INFO:teuthology.orchestra.run.smithi116.stdout:(5/120): ceph-mds-18.0.0-7537.gbcbf278c.el9.x86 3.6 MB/s | 2.1 MB 00:00 2023-12-03T11:40:33.461 INFO:teuthology.orchestra.run.smithi152.stdout:(12/120): ceph-radosgw-18.0.0-7537.gbcbf278c.el 4.1 MB/s | 8.6 MB 00:02 2023-12-03T11:40:33.486 INFO:teuthology.orchestra.run.smithi116.stdout:(6/120): ceph-mgr-18.0.0-7537.gbcbf278c.el9.x86 2.8 MB/s | 1.5 MB 00:00 2023-12-03T11:40:33.543 INFO:teuthology.orchestra.run.smithi152.stdout:(13/120): libcephsqlite-18.0.0-7537.gbcbf278c.e 1.9 MB/s | 164 kB 00:00 2023-12-03T11:40:33.593 INFO:teuthology.orchestra.run.smithi152.stdout:(14/120): libcephfs2-18.0.0-7537.gbcbf278c.el9. 1.1 MB/s | 718 kB 00:00 2023-12-03T11:40:33.677 INFO:teuthology.orchestra.run.smithi152.stdout:(15/120): librados-devel-18.0.0-7537.gbcbf278c. 946 kB/s | 126 kB 00:00 2023-12-03T11:40:34.152 INFO:teuthology.orchestra.run.smithi152.stdout:(16/120): libradosstriper1-18.0.0-7537.gbcbf278 851 kB/s | 475 kB 00:00 2023-12-03T11:40:34.227 INFO:teuthology.orchestra.run.smithi152.stdout:(17/120): python3-ceph-argparse-18.0.0-7537.gbc 602 kB/s | 45 kB 00:00 2023-12-03T11:40:34.369 INFO:teuthology.orchestra.run.smithi152.stdout:(18/120): python3-ceph-common-18.0.0-7537.gbcbf 910 kB/s | 129 kB 00:00 2023-12-03T11:40:34.520 INFO:teuthology.orchestra.run.smithi152.stdout:(19/120): python3-cephfs-18.0.0-7537.gbcbf278c. 1.1 MB/s | 162 kB 00:00 2023-12-03T11:40:34.603 INFO:teuthology.orchestra.run.smithi152.stdout:(20/120): librgw2-18.0.0-7537.gbcbf278c.el9.x86 5.2 MB/s | 4.8 MB 00:00 2023-12-03T11:40:34.691 INFO:teuthology.orchestra.run.smithi116.stdout:(7/120): ceph-mon-18.0.0-7537.gbcbf278c.el9.x86 3.4 MB/s | 4.4 MB 00:01 2023-12-03T11:40:34.704 INFO:teuthology.orchestra.run.smithi152.stdout:(21/120): python3-rados-18.0.0-7537.gbcbf278c.e 1.7 MB/s | 320 kB 00:00 2023-12-03T11:40:34.737 INFO:teuthology.orchestra.run.smithi152.stdout:(22/120): python3-rbd-18.0.0-7537.gbcbf278c.el9 2.2 MB/s | 298 kB 00:00 2023-12-03T11:40:34.763 INFO:teuthology.orchestra.run.smithi152.stdout:(23/120): python3-rgw-18.0.0-7537.gbcbf278c.el9 1.7 MB/s | 99 kB 00:00 2023-12-03T11:40:34.830 INFO:teuthology.orchestra.run.smithi152.stdout:(24/120): rbd-fuse-18.0.0-7537.gbcbf278c.el9.x8 928 kB/s | 85 kB 00:00 2023-12-03T11:40:34.875 INFO:teuthology.orchestra.run.smithi116.stdout:(8/120): ceph-common-18.0.0-7537.gbcbf278c.el9. 7.0 MB/s | 19 MB 00:02 2023-12-03T11:40:34.897 INFO:teuthology.orchestra.run.smithi152.stdout:(25/120): rbd-nbd-18.0.0-7537.gbcbf278c.el9.x86 2.5 MB/s | 170 kB 00:00 2023-12-03T11:40:34.917 INFO:teuthology.orchestra.run.smithi116.stdout:(9/120): ceph-selinux-18.0.0-7537.gbcbf278c.el9 581 kB/s | 24 kB 00:00 2023-12-03T11:40:35.003 INFO:teuthology.orchestra.run.smithi152.stdout:(26/120): ceph-grafana-dashboards-18.0.0-7537.g 265 kB/s | 28 kB 00:00 2023-12-03T11:40:35.128 INFO:teuthology.orchestra.run.smithi152.stdout:(27/120): ceph-mgr-cephadm-18.0.0-7537.gbcbf278 1.0 MB/s | 133 kB 00:00 2023-12-03T11:40:35.512 INFO:teuthology.orchestra.run.smithi152.stdout:(28/120): ceph-mgr-dashboard-18.0.0-7537.gbcbf2 4.6 MB/s | 1.8 MB 00:00 2023-12-03T11:40:35.679 INFO:teuthology.orchestra.run.smithi152.stdout:(29/120): rbd-mirror-18.0.0-7537.gbcbf278c.el9. 3.3 MB/s | 3.0 MB 00:00 2023-12-03T11:40:35.771 INFO:teuthology.orchestra.run.smithi152.stdout:(30/120): ceph-mgr-modules-core-18.0.0-7537.gbc 2.6 MB/s | 243 kB 00:00 2023-12-03T11:40:35.871 INFO:teuthology.orchestra.run.smithi152.stdout:(31/120): ceph-mgr-rook-18.0.0-7537.gbcbf278c.e 503 kB/s | 50 kB 00:00 2023-12-03T11:40:35.963 INFO:teuthology.orchestra.run.smithi152.stdout:(32/120): ceph-prometheus-alerts-18.0.0-7537.gb 165 kB/s | 15 kB 00:00 2023-12-03T11:40:36.047 INFO:teuthology.orchestra.run.smithi152.stdout:(33/120): ceph-volume-18.0.0-7537.gbcbf278c.el9 3.1 MB/s | 262 kB 00:00 2023-12-03T11:40:36.142 INFO:teuthology.orchestra.run.smithi116.stdout:(10/120): ceph-osd-18.0.0-7537.gbcbf278c.el9.x8 6.7 MB/s | 18 MB 00:02 2023-12-03T11:40:36.226 INFO:teuthology.orchestra.run.smithi116.stdout:(11/120): libcephfs-devel-18.0.0-7537.gbcbf278c 373 kB/s | 31 kB 00:00 2023-12-03T11:40:36.247 INFO:teuthology.orchestra.run.smithi152.stdout:(34/120): cephadm-18.0.0-7537.gbcbf278c.el9.noa 3.8 MB/s | 774 kB 00:00 2023-12-03T11:40:36.356 INFO:teuthology.orchestra.run.smithi152.stdout:(35/120): boost-program-options-1.75.0-8.el9.x8 976 kB/s | 106 kB 00:00 2023-12-03T11:40:36.376 INFO:teuthology.orchestra.run.smithi116.stdout:(12/120): libcephfs2-18.0.0-7537.gbcbf278c.el9. 4.7 MB/s | 718 kB 00:00 2023-12-03T11:40:36.389 INFO:teuthology.orchestra.run.smithi152.stdout:(36/120): flexiblas-3.0.4-8.el9.x86_64.rpm 954 kB/s | 32 kB 00:00 2023-12-03T11:40:36.451 INFO:teuthology.orchestra.run.smithi116.stdout:(13/120): libcephsqlite-18.0.0-7537.gbcbf278c.e 2.1 MB/s | 164 kB 00:00 2023-12-03T11:40:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:(14/120): librados-devel-18.0.0-7537.gbcbf278c. 1.9 MB/s | 126 kB 00:00 2023-12-03T11:40:36.523 INFO:teuthology.orchestra.run.smithi152.stdout:(37/120): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2023-12-03T11:40:36.557 INFO:teuthology.orchestra.run.smithi152.stdout:(38/120): flexiblas-openblas-openmp-3.0.4-8.el9 518 kB/s | 17 kB 00:00 2023-12-03T11:40:36.582 INFO:teuthology.orchestra.run.smithi152.stdout:(39/120): jq-1.6-16.el9.x86_64.rpm 7.3 MB/s | 186 kB 00:00 2023-12-03T11:40:36.616 INFO:teuthology.orchestra.run.smithi152.stdout:(40/120): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2023-12-03T11:40:36.660 INFO:teuthology.orchestra.run.smithi116.stdout:(15/120): libradosstriper1-18.0.0-7537.gbcbf278 3.3 MB/s | 475 kB 00:00 2023-12-03T11:40:36.708 INFO:teuthology.orchestra.run.smithi152.stdout:(41/120): ceph-mgr-diskprediction-local-18.0.0- 6.2 MB/s | 7.4 MB 00:01 2023-12-03T11:40:36.725 INFO:teuthology.orchestra.run.smithi152.stdout:(42/120): librabbitmq-0.11.0-7.el9.x86_64.rpm 407 kB/s | 44 kB 00:00 2023-12-03T11:40:36.758 INFO:teuthology.orchestra.run.smithi152.stdout:(43/120): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-03T11:40:36.784 INFO:teuthology.orchestra.run.smithi152.stdout:(44/120): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-12-03T11:40:36.801 INFO:teuthology.orchestra.run.smithi152.stdout:(45/120): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-12-03T11:40:36.826 INFO:teuthology.orchestra.run.smithi152.stdout:(46/120): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-03T11:40:36.851 INFO:teuthology.orchestra.run.smithi152.stdout:(47/120): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2023-12-03T11:40:36.968 INFO:teuthology.orchestra.run.smithi152.stdout:(48/120): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2023-12-03T11:40:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:(16/120): ceph-radosgw-18.0.0-7537.gbcbf278c.el 3.6 MB/s | 8.6 MB 00:02 2023-12-03T11:40:37.094 INFO:teuthology.orchestra.run.smithi152.stdout:(49/120): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2023-12-03T11:40:37.119 INFO:teuthology.orchestra.run.smithi152.stdout:(50/120): python3-jinja2-2.11.3-4.el9.noarch.rp 1.5 MB/s | 228 kB 00:00 2023-12-03T11:40:37.135 INFO:teuthology.orchestra.run.smithi116.stdout:(17/120): python3-ceph-argparse-18.0.0-7537.gbc 603 kB/s | 45 kB 00:00 2023-12-03T11:40:37.144 INFO:teuthology.orchestra.run.smithi152.stdout:(51/120): python3-jsonpatch-1.21-16.el9.noarch. 491 kB/s | 24 kB 00:00 2023-12-03T11:40:37.170 INFO:teuthology.orchestra.run.smithi152.stdout:(52/120): python3-jsonpointer-2.0-4.el9.noarch. 359 kB/s | 18 kB 00:00 2023-12-03T11:40:37.195 INFO:teuthology.orchestra.run.smithi152.stdout:(53/120): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-03T11:40:37.202 INFO:teuthology.orchestra.run.smithi116.stdout:(18/120): python3-ceph-common-18.0.0-7537.gbcbf 1.9 MB/s | 129 kB 00:00 2023-12-03T11:40:37.220 INFO:teuthology.orchestra.run.smithi152.stdout:(54/120): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2023-12-03T11:40:37.269 INFO:teuthology.orchestra.run.smithi116.stdout:(19/120): librgw2-18.0.0-7537.gbcbf278c.el9.x86 7.8 MB/s | 4.8 MB 00:00 2023-12-03T11:40:37.294 INFO:teuthology.orchestra.run.smithi116.stdout:(20/120): python3-cephfs-18.0.0-7537.gbcbf278c. 1.7 MB/s | 162 kB 00:00 2023-12-03T11:40:37.328 INFO:teuthology.orchestra.run.smithi116.stdout:(21/120): python3-rados-18.0.0-7537.gbcbf278c.e 5.4 MB/s | 320 kB 00:00 2023-12-03T11:40:37.329 INFO:teuthology.orchestra.run.smithi152.stdout:(55/120): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2023-12-03T11:40:37.354 INFO:teuthology.orchestra.run.smithi152.stdout:(56/120): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2023-12-03T11:40:37.380 INFO:teuthology.orchestra.run.smithi152.stdout:(57/120): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2023-12-03T11:40:37.387 INFO:teuthology.orchestra.run.smithi116.stdout:(22/120): python3-rbd-18.0.0-7537.gbcbf278c.el9 3.2 MB/s | 298 kB 00:00 2023-12-03T11:40:37.404 INFO:teuthology.orchestra.run.smithi152.stdout:(58/120): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2023-12-03T11:40:37.412 INFO:teuthology.orchestra.run.smithi116.stdout:(23/120): python3-rgw-18.0.0-7537.gbcbf278c.el9 1.2 MB/s | 99 kB 00:00 2023-12-03T11:40:37.430 INFO:teuthology.orchestra.run.smithi152.stdout:(59/120): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-12-03T11:40:37.437 INFO:teuthology.orchestra.run.smithi116.stdout:(24/120): rbd-fuse-18.0.0-7537.gbcbf278c.el9.x8 1.7 MB/s | 85 kB 00:00 2023-12-03T11:40:37.455 INFO:teuthology.orchestra.run.smithi152.stdout:(60/120): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-03T11:40:37.480 INFO:teuthology.orchestra.run.smithi152.stdout:(61/120): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2023-12-03T11:40:37.504 INFO:teuthology.orchestra.run.smithi116.stdout:(25/120): rbd-nbd-18.0.0-7537.gbcbf278c.el9.x86 2.5 MB/s | 170 kB 00:00 2023-12-03T11:40:37.506 INFO:teuthology.orchestra.run.smithi152.stdout:(62/120): python3-requests-oauthlib-1.3.0-12.el 847 kB/s | 43 kB 00:00 2023-12-03T11:40:37.563 INFO:teuthology.orchestra.run.smithi116.stdout:(26/120): ceph-grafana-dashboards-18.0.0-7537.g 482 kB/s | 28 kB 00:00 2023-12-03T11:40:37.638 INFO:teuthology.orchestra.run.smithi116.stdout:(27/120): ceph-mgr-cephadm-18.0.0-7537.gbcbf278 1.7 MB/s | 133 kB 00:00 2023-12-03T11:40:37.648 INFO:teuthology.orchestra.run.smithi152.stdout:(63/120): python3-toml-0.10.2-6.el9.noarch.rpm 278 kB/s | 39 kB 00:00 2023-12-03T11:40:37.797 INFO:teuthology.orchestra.run.smithi116.stdout:(28/120): rbd-mirror-18.0.0-7537.gbcbf278c.el9. 7.8 MB/s | 3.0 MB 00:00 2023-12-03T11:40:37.824 INFO:teuthology.orchestra.run.smithi152.stdout:(64/120): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2023-12-03T11:40:37.849 INFO:teuthology.orchestra.run.smithi152.stdout:(65/120): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2023-12-03T11:40:37.866 INFO:teuthology.orchestra.run.smithi152.stdout:(66/120): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-03T11:40:37.883 INFO:teuthology.orchestra.run.smithi152.stdout:(67/120): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2023-12-03T11:40:37.925 INFO:teuthology.orchestra.run.smithi152.stdout:(68/120): libgfortran-11.4.1-2.3.el9.x86_64.rpm 13 MB/s | 803 kB 00:00 2023-12-03T11:40:37.950 INFO:teuthology.orchestra.run.smithi152.stdout:(69/120): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2023-12-03T11:40:37.975 INFO:teuthology.orchestra.run.smithi152.stdout:(70/120): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-03T11:40:38.042 INFO:teuthology.orchestra.run.smithi152.stdout:(71/120): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-12-03T11:40:38.055 INFO:teuthology.orchestra.run.smithi116.stdout:(29/120): ceph-mgr-dashboard-18.0.0-7537.gbcbf2 4.2 MB/s | 1.8 MB 00:00 2023-12-03T11:40:38.076 INFO:teuthology.orchestra.run.smithi152.stdout:(72/120): python3-cryptography-36.0.1-4.el9.x86 9.1 MB/s | 1.1 MB 00:00 2023-12-03T11:40:38.101 INFO:teuthology.orchestra.run.smithi152.stdout:(73/120): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2023-12-03T11:40:38.126 INFO:teuthology.orchestra.run.smithi152.stdout:(74/120): python3-requests-2.25.1-7.el9.noarch. 2.2 MB/s | 113 kB 00:00 2023-12-03T11:40:38.152 INFO:teuthology.orchestra.run.smithi152.stdout:(75/120): python3-urllib3-1.26.5-4.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2023-12-03T11:40:38.172 INFO:teuthology.orchestra.run.smithi116.stdout:(30/120): ceph-mgr-modules-core-18.0.0-7537.gbc 2.0 MB/s | 243 kB 00:00 2023-12-03T11:40:38.240 INFO:teuthology.orchestra.run.smithi116.stdout:(31/120): ceph-mgr-rook-18.0.0-7537.gbcbf278c.e 751 kB/s | 50 kB 00:00 2023-12-03T11:40:38.298 INFO:teuthology.orchestra.run.smithi116.stdout:(32/120): ceph-prometheus-alerts-18.0.0-7537.gb 260 kB/s | 15 kB 00:00 2023-12-03T11:40:38.415 INFO:teuthology.orchestra.run.smithi116.stdout:(33/120): ceph-volume-18.0.0-7537.gbcbf278c.el9 2.2 MB/s | 262 kB 00:00 2023-12-03T11:40:38.649 INFO:teuthology.orchestra.run.smithi116.stdout:(34/120): cephadm-18.0.0-7537.gbcbf278c.el9.noa 3.2 MB/s | 774 kB 00:00 2023-12-03T11:40:38.732 INFO:teuthology.orchestra.run.smithi116.stdout:(35/120): ceph-mgr-diskprediction-local-18.0.0- 7.9 MB/s | 7.4 MB 00:00 2023-12-03T11:40:38.744 INFO:teuthology.orchestra.run.smithi152.stdout:(76/120): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2023-12-03T11:40:38.772 INFO:teuthology.orchestra.run.smithi152.stdout:(77/120): python3-devel-3.9.17-2.el9.x86_64.rpm 380 kB/s | 245 kB 00:00 2023-12-03T11:40:38.841 INFO:teuthology.orchestra.run.smithi116.stdout:(36/120): flexiblas-3.0.4-8.el9.x86_64.rpm 293 kB/s | 32 kB 00:00 2023-12-03T11:40:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:(37/120): boost-program-options-1.75.0-8.el9.x8 487 kB/s | 106 kB 00:00 2023-12-03T11:40:38.925 INFO:teuthology.orchestra.run.smithi116.stdout:(38/120): flexiblas-openblas-openmp-3.0.4-8.el9 297 kB/s | 17 kB 00:00 2023-12-03T11:40:38.975 INFO:teuthology.orchestra.run.smithi116.stdout:(39/120): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2023-12-03T11:40:39.000 INFO:teuthology.orchestra.run.smithi116.stdout:(40/120): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2023-12-03T11:40:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:(41/120): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-03T11:40:39.051 INFO:teuthology.orchestra.run.smithi116.stdout:(42/120): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2023-12-03T11:40:39.076 INFO:teuthology.orchestra.run.smithi116.stdout:(43/120): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-03T11:40:39.101 INFO:teuthology.orchestra.run.smithi116.stdout:(44/120): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2023-12-03T11:40:39.127 INFO:teuthology.orchestra.run.smithi116.stdout:(45/120): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-03T11:40:39.152 INFO:teuthology.orchestra.run.smithi116.stdout:(46/120): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-03T11:40:39.177 INFO:teuthology.orchestra.run.smithi116.stdout:(47/120): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2023-12-03T11:40:39.457 INFO:teuthology.orchestra.run.smithi152.stdout:(78/120): ceph-test-18.0.0-7537.gbcbf278c.el9.x 6.2 MB/s | 41 MB 00:06 2023-12-03T11:40:39.561 INFO:teuthology.orchestra.run.smithi116.stdout:(48/120): ceph-test-18.0.0-7537.gbcbf278c.el9.x 8.9 MB/s | 41 MB 00:04 2023-12-03T11:40:39.636 INFO:teuthology.orchestra.run.smithi116.stdout:(49/120): openblas-openmp-0.3.21-2.el9.x86_64.r 9.8 MB/s | 4.7 MB 00:00 2023-12-03T11:40:39.728 INFO:teuthology.orchestra.run.smithi116.stdout:(50/120): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2023-12-03T11:40:39.753 INFO:teuthology.orchestra.run.smithi116.stdout:(51/120): python3-jinja2-2.11.3-4.el9.noarch.rp 1.2 MB/s | 228 kB 00:00 2023-12-03T11:40:39.779 INFO:teuthology.orchestra.run.smithi116.stdout:(52/120): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2023-12-03T11:40:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:(53/120): python3-jsonpointer-2.0-4.el9.noarch. 242 kB/s | 18 kB 00:00 2023-12-03T11:40:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:(54/120): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2023-12-03T11:40:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:(55/120): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2023-12-03T11:40:39.906 INFO:teuthology.orchestra.run.smithi152.stdout:(79/120): libarrow-doc-9.0.0-9.el9.noarch.rpm 54 kB/s | 24 kB 00:00 2023-12-03T11:40:39.930 INFO:teuthology.orchestra.run.smithi116.stdout:(56/120): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2023-12-03T11:40:39.931 INFO:teuthology.orchestra.run.smithi152.stdout:(80/120): gperftools-libs-2.9.1-2.el9.x86_64.rp 260 kB/s | 309 kB 00:01 2023-12-03T11:40:39.955 INFO:teuthology.orchestra.run.smithi116.stdout:(57/120): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2023-12-03T11:40:39.980 INFO:teuthology.orchestra.run.smithi116.stdout:(58/120): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2023-12-03T11:40:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:(59/120): python3-prettytable-0.7.2-27.el9.noar 622 kB/s | 41 kB 00:00 2023-12-03T11:40:40.022 INFO:teuthology.orchestra.run.smithi116.stdout:(60/120): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2023-12-03T11:40:40.048 INFO:teuthology.orchestra.run.smithi116.stdout:(61/120): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 211 kB 00:00 2023-12-03T11:40:40.074 INFO:teuthology.orchestra.run.smithi116.stdout:(62/120): python3-pytz-2021.1-5.el9.noarch.rpm 620 kB/s | 47 kB 00:00 2023-12-03T11:40:40.090 INFO:teuthology.orchestra.run.smithi152.stdout:(81/120): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2023-12-03T11:40:40.107 INFO:teuthology.orchestra.run.smithi116.stdout:(63/120): python3-requests-oauthlib-1.3.0-12.el 732 kB/s | 43 kB 00:00 2023-12-03T11:40:40.207 INFO:teuthology.orchestra.run.smithi152.stdout:(82/120): liboath-2.6.7-2.el9.x86_64.rpm 162 kB/s | 49 kB 00:00 2023-12-03T11:40:40.315 INFO:teuthology.orchestra.run.smithi152.stdout:(83/120): luarocks-3.9.2-1.el9.noarch.rpm 671 kB/s | 151 kB 00:00 2023-12-03T11:40:40.375 INFO:teuthology.orchestra.run.smithi116.stdout:(64/120): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2023-12-03T11:40:40.400 INFO:teuthology.orchestra.run.smithi116.stdout:(65/120): python3-toml-0.10.2-6.el9.noarch.rpm 120 kB/s | 39 kB 00:00 2023-12-03T11:40:40.425 INFO:teuthology.orchestra.run.smithi116.stdout:(66/120): socat-1.7.4.1-5.el9.x86_64.rpm 955 kB/s | 300 kB 00:00 2023-12-03T11:40:40.450 INFO:teuthology.orchestra.run.smithi116.stdout:(67/120): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2023-12-03T11:40:40.467 INFO:teuthology.orchestra.run.smithi116.stdout:(68/120): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2023-12-03T11:40:40.501 INFO:teuthology.orchestra.run.smithi116.stdout:(69/120): libgfortran-11.4.1-2.3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2023-12-03T11:40:40.526 INFO:teuthology.orchestra.run.smithi116.stdout:(70/120): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2023-12-03T11:40:40.559 INFO:teuthology.orchestra.run.smithi116.stdout:(71/120): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-03T11:40:40.591 INFO:teuthology.orchestra.run.smithi152.stdout:(84/120): libarrow-9.0.0-9.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2023-12-03T11:40:40.593 INFO:teuthology.orchestra.run.smithi116.stdout:(72/120): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2023-12-03T11:40:40.618 INFO:teuthology.orchestra.run.smithi116.stdout:(73/120): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-12-03T11:40:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:(85/120): python3-asyncssh-2.13.2-3.el9.noarch. 1.7 MB/s | 548 kB 00:00 2023-12-03T11:40:40.644 INFO:teuthology.orchestra.run.smithi116.stdout:(74/120): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2023-12-03T11:40:40.669 INFO:teuthology.orchestra.run.smithi116.stdout:(75/120): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-12-03T11:40:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:(76/120): python3-urllib3-1.26.5-4.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2023-12-03T11:40:40.741 INFO:teuthology.orchestra.run.smithi152.stdout:(86/120): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2023-12-03T11:40:40.775 INFO:teuthology.orchestra.run.smithi152.stdout:(87/120): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.4 MB/s | 837 kB 00:00 2023-12-03T11:40:40.791 INFO:teuthology.orchestra.run.smithi152.stdout:(88/120): python3-cachetools-4.2.4-1.el9.noarch 193 kB/s | 32 kB 00:00 2023-12-03T11:40:40.892 INFO:teuthology.orchestra.run.smithi152.stdout:(89/120): python3-certifi-2021.10.8-2.el9.noarc 99 kB/s | 15 kB 00:00 2023-12-03T11:40:40.934 INFO:teuthology.orchestra.run.smithi152.stdout:(90/120): python3-cheroot-8.6.0-4.el9.noarch.rp 1.1 MB/s | 172 kB 00:00 2023-12-03T11:40:40.968 INFO:teuthology.orchestra.run.smithi152.stdout:(91/120): python3-cherrypy-18.6.1-2.el9.noarch. 2.0 MB/s | 358 kB 00:00 2023-12-03T11:40:41.068 INFO:teuthology.orchestra.run.smithi152.stdout:(92/120): python3-google-auth-2.23.4-1.el9.noar 1.1 MB/s | 197 kB 00:00 2023-12-03T11:40:41.078 INFO:teuthology.orchestra.run.smithi116.stdout:(77/120): lua-devel-5.4.4-4.el9.x86_64.rpm 55 kB/s | 22 kB 00:00 2023-12-03T11:40:41.085 INFO:teuthology.orchestra.run.smithi152.stdout:(93/120): python3-jaraco-8.2.1-3.el9.noarch.rpm 71 kB/s | 11 kB 00:00 2023-12-03T11:40:41.118 INFO:teuthology.orchestra.run.smithi152.stdout:(94/120): python3-jaraco-classes-3.2.1-5.el9.no 117 kB/s | 18 kB 00:00 2023-12-03T11:40:41.218 INFO:teuthology.orchestra.run.smithi152.stdout:(95/120): python3-jaraco-collections-3.0.0-8.el 154 kB/s | 23 kB 00:00 2023-12-03T11:40:41.236 INFO:teuthology.orchestra.run.smithi152.stdout:(96/120): python3-jaraco-functools-3.5.0-2.el9. 129 kB/s | 19 kB 00:00 2023-12-03T11:40:41.269 INFO:teuthology.orchestra.run.smithi152.stdout:(97/120): python3-jaraco-text-3.2.0-6.el9.noarc 131 kB/s | 20 kB 00:00 2023-12-03T11:40:41.286 INFO:teuthology.orchestra.run.smithi116.stdout:(78/120): python3-devel-3.9.17-2.el9.x86_64.rpm 398 kB/s | 245 kB 00:00 2023-12-03T11:40:41.369 INFO:teuthology.orchestra.run.smithi152.stdout:(98/120): python3-jwt+crypto-2.4.0-1.el9.noarch 60 kB/s | 9.0 kB 00:00 2023-12-03T11:40:41.378 INFO:teuthology.orchestra.run.smithi116.stdout:(79/120): gperftools-libs-2.9.1-2.el9.x86_64.rp 452 kB/s | 309 kB 00:00 2023-12-03T11:40:41.386 INFO:teuthology.orchestra.run.smithi152.stdout:(99/120): python3-jwt-2.4.0-1.el9.noarch.rpm 272 kB/s | 41 kB 00:00 2023-12-03T11:40:41.495 INFO:teuthology.orchestra.run.smithi116.stdout:(80/120): libarrow-doc-9.0.0-9.el9.noarch.rpm 117 kB/s | 24 kB 00:00 2023-12-03T11:40:41.537 INFO:teuthology.orchestra.run.smithi152.stdout:(100/120): python3-logutils-0.3.5-21.el9.noarch 277 kB/s | 46 kB 00:00 2023-12-03T11:40:41.562 INFO:teuthology.orchestra.run.smithi152.stdout:(101/120): python3-more-itertools-8.12.0-2.el9. 450 kB/s | 79 kB 00:00 2023-12-03T11:40:41.595 INFO:teuthology.orchestra.run.smithi152.stdout:(102/120): python3-kubernetes-26.1.0-2.el9.noar 3.1 MB/s | 1.0 MB 00:00 2023-12-03T11:40:41.620 INFO:teuthology.orchestra.run.smithi116.stdout:(81/120): liboath-2.6.7-2.el9.x86_64.rpm 201 kB/s | 49 kB 00:00 2023-12-03T11:40:41.687 INFO:teuthology.orchestra.run.smithi116.stdout:(82/120): libarrow-9.0.0-9.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2023-12-03T11:40:41.688 INFO:teuthology.orchestra.run.smithi152.stdout:(103/120): python3-natsort-7.1.1-5.el9.noarch.r 383 kB/s | 58 kB 00:00 2023-12-03T11:40:41.713 INFO:teuthology.orchestra.run.smithi116.stdout:(83/120): libunwind-1.6.2-1.el9.x86_64.rpm 311 kB/s | 67 kB 00:00 2023-12-03T11:40:41.720 INFO:teuthology.orchestra.run.smithi152.stdout:(104/120): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2023-12-03T11:40:41.738 INFO:teuthology.orchestra.run.smithi116.stdout:(84/120): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2023-12-03T11:40:41.746 INFO:teuthology.orchestra.run.smithi152.stdout:(105/120): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2023-12-03T11:40:41.805 INFO:teuthology.orchestra.run.smithi116.stdout:(85/120): parquet-libs-9.0.0-9.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2023-12-03T11:40:41.830 INFO:teuthology.orchestra.run.smithi116.stdout:(86/120): python3-asyncssh-2.13.2-3.el9.noarch. 4.6 MB/s | 548 kB 00:00 2023-12-03T11:40:41.846 INFO:teuthology.orchestra.run.smithi152.stdout:(106/120): python3-pyOpenSSL-21.0.0-1.el9.noarc 567 kB/s | 90 kB 00:00 2023-12-03T11:40:41.847 INFO:teuthology.orchestra.run.smithi116.stdout:(87/120): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2023-12-03T11:40:41.872 INFO:teuthology.orchestra.run.smithi152.stdout:(107/120): python3-repoze-lru-0.7-16.el9.noarch 204 kB/s | 31 kB 00:00 2023-12-03T11:40:41.897 INFO:teuthology.orchestra.run.smithi116.stdout:(88/120): python3-cachetools-4.2.4-1.el9.noarch 353 kB/s | 32 kB 00:00 2023-12-03T11:40:41.922 INFO:teuthology.orchestra.run.smithi116.stdout:(89/120): python3-certifi-2021.10.8-2.el9.noarc 163 kB/s | 15 kB 00:00 2023-12-03T11:40:41.938 INFO:teuthology.orchestra.run.smithi152.stdout:(108/120): python3-routes-2.5.1-5.el9.noarch.rp 981 kB/s | 188 kB 00:00 2023-12-03T11:40:41.956 INFO:teuthology.orchestra.run.smithi116.stdout:(90/120): python3-cheroot-8.6.0-4.el9.noarch.rp 1.5 MB/s | 172 kB 00:00 2023-12-03T11:40:42.005 INFO:teuthology.orchestra.run.smithi152.stdout:(109/120): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2023-12-03T11:40:42.006 INFO:teuthology.orchestra.run.smithi116.stdout:(91/120): python3-cherrypy-18.6.1-2.el9.noarch. 3.2 MB/s | 358 kB 00:00 2023-12-03T11:40:42.030 INFO:teuthology.orchestra.run.smithi152.stdout:(110/120): python3-tempora-5.0.0-2.el9.noarch.r 225 kB/s | 36 kB 00:00 2023-12-03T11:40:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:(92/120): python3-google-auth-2.23.4-1.el9.noar 1.8 MB/s | 197 kB 00:00 2023-12-03T11:40:42.049 INFO:teuthology.orchestra.run.smithi116.stdout:(93/120): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2023-12-03T11:40:42.097 INFO:teuthology.orchestra.run.smithi152.stdout:(111/120): python3-typing-extensions-4.4.0-2.el 321 kB/s | 51 kB 00:00 2023-12-03T11:40:42.098 INFO:teuthology.orchestra.run.smithi116.stdout:(94/120): python3-jaraco-classes-3.2.1-5.el9.no 192 kB/s | 18 kB 00:00 2023-12-03T11:40:42.124 INFO:teuthology.orchestra.run.smithi116.stdout:(95/120): python3-jaraco-collections-3.0.0-8.el 250 kB/s | 23 kB 00:00 2023-12-03T11:40:42.141 INFO:teuthology.orchestra.run.smithi116.stdout:(96/120): python3-jaraco-functools-3.5.0-2.el9. 210 kB/s | 19 kB 00:00 2023-12-03T11:40:42.164 INFO:teuthology.orchestra.run.smithi152.stdout:(112/120): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2023-12-03T11:40:42.181 INFO:teuthology.orchestra.run.smithi152.stdout:(113/120): python3-websocket-client-1.2.3-2.el9 596 kB/s | 90 kB 00:00 2023-12-03T11:40:42.199 INFO:teuthology.orchestra.run.smithi116.stdout:(97/120): python3-jaraco-text-3.2.0-6.el9.noarc 195 kB/s | 20 kB 00:00 2023-12-03T11:40:42.225 INFO:teuthology.orchestra.run.smithi116.stdout:(98/120): python3-jwt+crypto-2.4.0-1.el9.noarch 90 kB/s | 9.0 kB 00:00 2023-12-03T11:40:42.250 INFO:teuthology.orchestra.run.smithi116.stdout:(99/120): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2023-12-03T11:40:42.315 INFO:teuthology.orchestra.run.smithi152.stdout:(114/120): python3-werkzeug-2.0.3-3.el9.1.noarc 1.9 MB/s | 427 kB 00:00 2023-12-03T11:40:42.325 INFO:teuthology.orchestra.run.smithi116.stdout:(100/120): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2023-12-03T11:40:42.340 INFO:teuthology.orchestra.run.smithi152.stdout:(115/120): python3-zc-lockfile-2.0-10.el9.noarc 114 kB/s | 20 kB 00:00 2023-12-03T11:40:42.351 INFO:teuthology.orchestra.run.smithi116.stdout:(101/120): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2023-12-03T11:40:42.365 INFO:teuthology.orchestra.run.smithi152.stdout:(116/120): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2023-12-03T11:40:42.376 INFO:teuthology.orchestra.run.smithi116.stdout:(102/120): python3-more-itertools-8.12.0-2.el9. 628 kB/s | 79 kB 00:00 2023-12-03T11:40:42.426 INFO:teuthology.orchestra.run.smithi116.stdout:(103/120): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2023-12-03T11:40:42.460 INFO:teuthology.orchestra.run.smithi116.stdout:(104/120): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2023-12-03T11:40:42.485 INFO:teuthology.orchestra.run.smithi116.stdout:(105/120): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2023-12-03T11:40:42.527 INFO:teuthology.orchestra.run.smithi116.stdout:(106/120): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2023-12-03T11:40:42.540 INFO:teuthology.orchestra.run.smithi152.stdout:(117/120): s3cmd-2.3.0-4.el9.noarch.rpm 873 kB/s | 197 kB 00:00 2023-12-03T11:40:42.561 INFO:teuthology.orchestra.run.smithi116.stdout:(107/120): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2023-12-03T11:40:42.616 INFO:teuthology.orchestra.run.smithi152.stdout:(118/120): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2023-12-03T11:40:42.644 INFO:teuthology.orchestra.run.smithi116.stdout:(108/120): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2023-12-03T11:40:42.850 INFO:teuthology.orchestra.run.smithi152.stdout:(119/120): librados2-18.0.0-7537.gbcbf278c.el9. 6.9 MB/s | 3.3 MB 00:00 2023-12-03T11:40:42.870 INFO:teuthology.orchestra.run.smithi116.stdout:(109/120): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2023-12-03T11:40:42.895 INFO:teuthology.orchestra.run.smithi116.stdout:(110/120): python3-tempora-5.0.0-2.el9.noarch.r 107 kB/s | 36 kB 00:00 2023-12-03T11:40:42.921 INFO:teuthology.orchestra.run.smithi116.stdout:(111/120): python3-typing-extensions-4.4.0-2.el 185 kB/s | 51 kB 00:00 2023-12-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi152.stdout:(120/120): librbd1-18.0.0-7537.gbcbf278c.el9.x8 7.2 MB/s | 3.0 MB 00:00 2023-12-03T11:40:42.962 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:40:42.962 INFO:teuthology.orchestra.run.smithi152.stdout:Total 14 MB/s | 181 MB 00:13 2023-12-03T11:40:42.985 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-03T11:40:42.995 INFO:teuthology.orchestra.run.smithi116.stdout:(112/120): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-03T11:40:43.021 INFO:teuthology.orchestra.run.smithi116.stdout:(113/120): python3-websocket-client-1.2.3-2.el9 717 kB/s | 90 kB 00:00 2023-12-03T11:40:43.054 INFO:teuthology.orchestra.run.smithi116.stdout:(114/120): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2023-12-03T11:40:43.084 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-03T11:40:43.084 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-03T11:40:43.096 INFO:teuthology.orchestra.run.smithi116.stdout:(115/120): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2023-12-03T11:40:43.122 INFO:teuthology.orchestra.run.smithi116.stdout:(116/120): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-12-03T11:40:43.155 INFO:teuthology.orchestra.run.smithi116.stdout:(117/120): s3cmd-2.3.0-4.el9.noarch.rpm 1.9 MB/s | 197 kB 00:00 2023-12-03T11:40:43.230 INFO:teuthology.orchestra.run.smithi116.stdout:(118/120): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-12-03T11:40:43.489 INFO:teuthology.orchestra.run.smithi116.stdout:(119/120): librbd1-18.0.0-7537.gbcbf278c.el9.x8 9.0 MB/s | 3.0 MB 00:00 2023-12-03T11:40:43.631 INFO:teuthology.orchestra.run.smithi116.stdout:(120/120): librados2-18.0.0-7537.gbcbf278c.el9. 6.5 MB/s | 3.3 MB 00:00 2023-12-03T11:40:43.638 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:40:43.638 INFO:teuthology.orchestra.run.smithi116.stdout:Total 15 MB/s | 181 MB 00:11 2023-12-03T11:40:43.662 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-12-03T11:40:43.760 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-12-03T11:40:43.761 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-12-03T11:40:44.582 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-03T11:40:44.583 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-03T11:40:45.282 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-12-03T11:40:45.283 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-12-03T11:40:49.494 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-03T11:40:49.569 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/122 2023-12-03T11:40:49.594 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/122 2023-12-03T11:40:49.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/122 2023-12-03T11:40:49.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/122 2023-12-03T11:40:49.729 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/122 2023-12-03T11:40:50.017 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-12-03T11:40:50.056 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 6/122 2023-12-03T11:40:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/122 2023-12-03T11:40:50.102 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/122 2023-12-03T11:40:50.121 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:40:50.147 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/122 2023-12-03T11:40:50.179 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/122 2023-12-03T11:40:50.223 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:40:50.231 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/122 2023-12-03T11:40:50.246 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:40:50.338 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:40:50.378 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:40:50.406 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:40:50.491 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:40:50.515 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:40:50.559 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 6/122 2023-12-03T11:40:50.612 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:40:50.624 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:40:50.661 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 12/122 2023-12-03T11:40:50.729 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 13/122 2023-12-03T11:40:50.732 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:40:50.763 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:40:50.792 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 14/122 2023-12-03T11:40:50.829 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 15/122 2023-12-03T11:40:50.867 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 16/122 2023-12-03T11:40:50.880 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:40:50.906 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 17/122 2023-12-03T11:40:50.920 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:40:50.948 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:40:50.970 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/122 2023-12-03T11:40:51.006 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/122 2023-12-03T11:40:51.033 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:40:51.039 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/122 2023-12-03T11:40:51.057 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:40:51.067 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 21/122 2023-12-03T11:40:51.136 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/122 2023-12-03T11:40:51.166 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:40:51.171 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 23/122 2023-12-03T11:40:51.203 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 12/122 2023-12-03T11:40:51.214 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 24/122 2023-12-03T11:40:51.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 25/122 2023-12-03T11:40:51.273 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 13/122 2023-12-03T11:40:51.313 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 26/122 2023-12-03T11:40:51.326 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 14/122 2023-12-03T11:40:51.363 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 15/122 2023-12-03T11:40:51.406 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 27/122 2023-12-03T11:40:51.408 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 16/122 2023-12-03T11:40:51.448 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 17/122 2023-12-03T11:40:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/122 2023-12-03T11:40:51.548 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/122 2023-12-03T11:40:51.549 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 28/122 2023-12-03T11:40:51.581 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/122 2023-12-03T11:40:51.601 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 29/122 2023-12-03T11:40:51.609 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 21/122 2023-12-03T11:40:51.678 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/122 2023-12-03T11:40:51.713 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 23/122 2023-12-03T11:40:51.757 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 24/122 2023-12-03T11:40:51.785 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 25/122 2023-12-03T11:40:51.857 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 26/122 2023-12-03T11:40:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 27/122 2023-12-03T11:40:52.076 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 28/122 2023-12-03T11:40:52.124 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 30/122 2023-12-03T11:40:52.136 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 29/122 2023-12-03T11:40:52.205 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 31/122 2023-12-03T11:40:52.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 32/122 2023-12-03T11:40:52.281 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 33/122 2023-12-03T11:40:52.312 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 34/122 2023-12-03T11:40:52.418 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 35/122 2023-12-03T11:40:52.456 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 36/122 2023-12-03T11:40:52.511 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 37/122 2023-12-03T11:40:52.584 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 38/122 2023-12-03T11:40:52.615 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 39/122 2023-12-03T11:40:52.652 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 40/122 2023-12-03T11:40:52.656 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 30/122 2023-12-03T11:40:52.699 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 41/122 2023-12-03T11:40:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 31/122 2023-12-03T11:40:52.744 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 42/122 2023-12-03T11:40:52.770 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 32/122 2023-12-03T11:40:52.785 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 43/122 2023-12-03T11:40:52.798 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 33/122 2023-12-03T11:40:52.829 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 34/122 2023-12-03T11:40:52.920 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 44/122 2023-12-03T11:40:52.935 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 35/122 2023-12-03T11:40:52.954 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 45/122 2023-12-03T11:40:52.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 36/122 2023-12-03T11:40:52.995 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 46/122 2023-12-03T11:40:53.021 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 37/122 2023-12-03T11:40:53.094 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 38/122 2023-12-03T11:40:53.097 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 47/122 2023-12-03T11:40:53.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 39/122 2023-12-03T11:40:53.129 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 48/122 2023-12-03T11:40:53.161 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 40/122 2023-12-03T11:40:53.170 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 49/122 2023-12-03T11:40:53.203 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 50/122 2023-12-03T11:40:53.207 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 41/122 2023-12-03T11:40:53.253 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 42/122 2023-12-03T11:40:53.295 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 43/122 2023-12-03T11:40:53.297 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 51/122 2023-12-03T11:40:53.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 52/122 2023-12-03T11:40:53.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 53/122 2023-12-03T11:40:53.430 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 44/122 2023-12-03T11:40:53.460 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 54/122 2023-12-03T11:40:53.472 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 45/122 2023-12-03T11:40:53.513 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 46/122 2023-12-03T11:40:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 47/122 2023-12-03T11:40:53.639 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 48/122 2023-12-03T11:40:53.660 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 55/122 2023-12-03T11:40:53.680 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 49/122 2023-12-03T11:40:53.709 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 56/122 2023-12-03T11:40:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 50/122 2023-12-03T11:40:53.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 51/122 2023-12-03T11:40:53.840 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 57/122 2023-12-03T11:40:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 52/122 2023-12-03T11:40:53.887 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 58/122 2023-12-03T11:40:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 53/122 2023-12-03T11:40:53.915 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 59/122 2023-12-03T11:40:53.949 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 60/122 2023-12-03T11:40:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 54/122 2023-12-03T11:40:53.982 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 61/122 2023-12-03T11:40:54.015 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 62/122 2023-12-03T11:40:54.068 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 63/122 2023-12-03T11:40:54.102 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 64/122 2023-12-03T11:40:54.132 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:40:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 55/122 2023-12-03T11:40:54.210 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 66/122 2023-12-03T11:40:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 56/122 2023-12-03T11:40:54.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 67/122 2023-12-03T11:40:54.276 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 68/122 2023-12-03T11:40:54.317 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 69/122 2023-12-03T11:40:54.352 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 70/122 2023-12-03T11:40:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 57/122 2023-12-03T11:40:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 58/122 2023-12-03T11:40:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 59/122 2023-12-03T11:40:54.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 71/122 2023-12-03T11:40:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 60/122 2023-12-03T11:40:54.479 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 72/122 2023-12-03T11:40:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 61/122 2023-12-03T11:40:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 62/122 2023-12-03T11:40:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 63/122 2023-12-03T11:40:54.582 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 73/122 2023-12-03T11:40:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 64/122 2023-12-03T11:40:54.630 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 74/122 2023-12-03T11:40:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:40:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 66/122 2023-12-03T11:40:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 67/122 2023-12-03T11:40:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 68/122 2023-12-03T11:40:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 69/122 2023-12-03T11:40:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 70/122 2023-12-03T11:40:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 71/122 2023-12-03T11:40:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 72/122 2023-12-03T11:40:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 73/122 2023-12-03T11:40:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 74/122 2023-12-03T11:40:55.373 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/122 2023-12-03T11:40:55.421 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 76/122 2023-12-03T11:40:55.452 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 77/122 2023-12-03T11:40:55.485 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 78/122 2023-12-03T11:40:55.663 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/122 2023-12-03T11:40:55.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/122 2023-12-03T11:40:55.761 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/122 2023-12-03T11:40:55.865 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/122 2023-12-03T11:40:55.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 76/122 2023-12-03T11:40:56.028 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 77/122 2023-12-03T11:40:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 78/122 2023-12-03T11:40:56.241 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/122 2023-12-03T11:40:56.267 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/122 2023-12-03T11:40:56.345 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/122 2023-12-03T11:40:56.421 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/122 2023-12-03T11:40:56.530 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/122 2023-12-03T11:40:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/122 2023-12-03T11:40:57.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/122 2023-12-03T11:40:57.797 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/122 2023-12-03T11:40:57.879 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/122 2023-12-03T11:40:57.910 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 86/122 2023-12-03T11:40:57.943 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 87/122 2023-12-03T11:40:58.237 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 88/122 2023-12-03T11:40:58.265 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:40:58.338 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:40:58.366 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/122 2023-12-03T11:40:58.368 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 90/122 2023-12-03T11:40:58.405 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 91/122 2023-12-03T11:40:58.442 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/122 2023-12-03T11:40:58.472 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 86/122 2023-12-03T11:40:58.506 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 87/122 2023-12-03T11:40:58.816 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 88/122 2023-12-03T11:40:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 92/122 2023-12-03T11:40:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:40:58.859 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:40:58.930 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:40:58.943 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:40:58.951 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 94/122 2023-12-03T11:40:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 90/122 2023-12-03T11:40:59.010 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 91/122 2023-12-03T11:40:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 92/122 2023-12-03T11:40:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:40:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:40:59.565 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 94/122 2023-12-03T11:41:01.132 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:01.174 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:01.656 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:01.734 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:01.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:41:01.862 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:41:01.862 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-03T11:41:01.862 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-03T11:41:01.862 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:01.903 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:41:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:41:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:41:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:41:02.376 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-03T11:41:02.376 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-03T11:41:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2023-12-03T11:41:13.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:41:13.502 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:41:13.502 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:13.502 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-03T11:41:13.502 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:13.649 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:41:13.649 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2023-12-03T11:41:13.649 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2023-12-03T11:41:13.649 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2023-12-03T11:41:13.649 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2023-12-03T11:41:13.650 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2023-12-03T11:41:13.650 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2023-12-03T11:41:13.650 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2023-12-03T11:41:13.650 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:13.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:41:13.825 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:41:13.825 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:13.825 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-03T11:41:13.825 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:15.080 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:41:15.107 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:15.315 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:41:15.337 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:41:15.337 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:15.337 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-03T11:41:15.337 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:41:15.338 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:41:15.338 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:41:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:41:15.431 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:15.431 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-03T11:41:15.432 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:41:15.432 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:41:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:41:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:15.782 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:41:15.805 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:16.098 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:41:16.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:16.204 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:41:16.239 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:41:16.269 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 103/122 2023-12-03T11:41:16.329 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 104/122 2023-12-03T11:41:16.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:41:16.569 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:41:16.575 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:41:16.605 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:41:16.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:41:16.634 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 103/122 2023-12-03T11:41:16.695 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 104/122 2023-12-03T11:41:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:41:16.939 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:41:16.970 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:41:17.566 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:41:17.596 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:41:17.696 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:41:17.853 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 108/122 2023-12-03T11:41:17.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:41:17.913 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:41:17.913 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:17.913 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-03T11:41:17.913 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:41:17.913 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:41:17.914 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:41:17.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:41:17.958 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:41:17.968 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:41:18.052 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:41:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 108/122 2023-12-03T11:41:18.239 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:41:18.262 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:41:18.262 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:18.262 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-03T11:41:18.263 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:41:18.263 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:41:18.263 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:18.307 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:41:18.318 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:41:18.735 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 111/122 2023-12-03T11:41:18.783 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:41:18.805 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:41:18.806 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:18.806 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-03T11:41:18.806 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:41:18.806 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:41:18.806 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:18.851 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:41:18.873 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:41:18.873 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:18.873 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-03T11:41:18.874 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:19.083 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 111/122 2023-12-03T11:41:19.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:41:19.138 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:41:19.139 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:19.139 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-03T11:41:19.139 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:41:19.139 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:41:19.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:19.177 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:41:19.199 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:19.200 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:41:19.222 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:41:19.222 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:19.222 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-03T11:41:19.223 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:19.517 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:41:19.539 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:41:19.540 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:41:19.540 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-03T11:41:19.540 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:41:19.540 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:41:19.540 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:23.021 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 115/122 2023-12-03T11:41:23.065 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 116/122 2023-12-03T11:41:23.161 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 117/122 2023-12-03T11:41:23.194 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 118/122 2023-12-03T11:41:23.244 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 119/122 2023-12-03T11:41:23.284 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : s3cmd-2.3.0-4.el9.noarch 120/122 2023-12-03T11:41:23.284 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:41:23.329 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 115/122 2023-12-03T11:41:23.368 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:41:23.368 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:23.372 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 116/122 2023-12-03T11:41:23.468 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 117/122 2023-12-03T11:41:23.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 118/122 2023-12-03T11:41:23.544 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 119/122 2023-12-03T11:41:23.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : s3cmd-2.3.0-4.el9.noarch 120/122 2023-12-03T11:41:23.576 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:41:23.659 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:41:23.659 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:27.677 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:27.677 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 1/122 2023-12-03T11:41:27.677 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 3/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 4/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 5/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 6/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 11/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 12/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 13/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 14/122 2023-12-03T11:41:27.678 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 15/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 16/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 17/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 18/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 19/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 20/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 21/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 22/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 23/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 24/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 25/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 26/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 27/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 28/122 2023-12-03T11:41:27.679 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 29/122 2023-12-03T11:41:27.687 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 30/122 2023-12-03T11:41:27.687 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 31/122 2023-12-03T11:41:27.687 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 32/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 33/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 34/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 35/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 36/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 37/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 38/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 39/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 40/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 41/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 42/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 43/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 44/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 45/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 46/122 2023-12-03T11:41:27.688 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 47/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 48/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 49/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 50/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 51/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 52/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 53/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 54/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 55/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 56/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 57/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 58/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 59/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 60/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 61/122 2023-12-03T11:41:27.689 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 62/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 63/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 64/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 66/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 67/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 68/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 69/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 70/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 71/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 72/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 73/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 74/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 75/122 2023-12-03T11:41:27.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 76/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 77/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 78/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 79/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 80/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 81/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 82/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 83/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 84/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 85/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 86/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 87/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 88/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 89/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 90/122 2023-12-03T11:41:27.691 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 91/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 92/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 93/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 94/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 95/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 96/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 97/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 98/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 99/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 100/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 101/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 102/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 103/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 104/122 2023-12-03T11:41:27.692 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 105/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 106/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 107/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 108/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 109/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 110/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 111/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 112/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 113/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 114/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 115/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 116/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : s3cmd-2.3.0-4.el9.noarch 117/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 118/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 119/122 2023-12-03T11:41:27.693 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 120/122 2023-12-03T11:41:27.694 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 121/122 2023-12-03T11:41:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 1/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 3/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 4/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 5/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 6/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:41:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 11/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 12/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 13/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 14/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 15/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 16/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 17/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 18/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 19/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 20/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 21/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 22/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 23/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 24/122 2023-12-03T11:41:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 25/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 26/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 27/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 28/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 29/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 30/122 2023-12-03T11:41:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 31/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 32/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 33/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 34/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 35/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 36/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 37/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 38/122 2023-12-03T11:41:27.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 39/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 40/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 41/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 42/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 43/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 44/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 45/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 46/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 47/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 48/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 49/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 50/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 51/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 52/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 53/122 2023-12-03T11:41:27.910 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 54/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 55/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 56/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 57/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 58/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 59/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 60/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 61/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 62/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 63/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 64/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 66/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 67/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 68/122 2023-12-03T11:41:27.911 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 69/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 70/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 71/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 72/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 73/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 74/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 75/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 76/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 77/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 78/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 79/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 80/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 81/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 82/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 83/122 2023-12-03T11:41:27.912 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 84/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 85/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 86/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 87/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 88/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 89/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 90/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 91/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 92/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 93/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 94/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 95/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 96/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 97/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 98/122 2023-12-03T11:41:27.913 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 99/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 100/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 101/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 102/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 103/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 104/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 105/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 106/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 107/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 108/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 109/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 110/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 111/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 112/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 113/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 114/122 2023-12-03T11:41:27.914 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 115/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 116/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : s3cmd-2.3.0-4.el9.noarch 117/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 118/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 119/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 120/122 2023-12-03T11:41:27.915 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 121/122 2023-12-03T11:41:29.921 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.922 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.923 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.924 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2023-12-03T11:41:29.925 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.926 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-03T11:41:29.927 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-03T11:41:29.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-03T11:41:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.930 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: s3cmd-2.3.0-4.el9.noarch 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:29.931 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-03T11:41:30.068 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:41:30.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:30.068 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-12-03T11:41:30.068 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.068 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-03T11:41:30.070 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.071 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-03T11:41:30.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-03T11:41:30.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-03T11:41:30.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-03T11:41:30.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-03T11:41:30.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: s3cmd-2.3.0-4.el9.noarch 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-03T11:41:30.077 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-03T11:41:30.078 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:30.078 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-12-03T11:41:30.221 DEBUG:teuthology.parallel:result is None 2023-12-03T11:41:30.381 DEBUG:teuthology.parallel:result is None 2023-12-03T11:41:30.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:41:30.647 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-03T11:41:30.668 INFO:teuthology.orchestra.run.smithi116.stdout:18.0.0-7537.gbcbf278c.el9 2023-12-03T11:41:30.669 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7537.gbcbf278c.el9 2023-12-03T11:41:30.669 INFO:teuthology.task.install:The correct ceph version 18.0.0-7537.gbcbf278c is installed. 2023-12-03T11:41:30.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:41:30.906 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-03T11:41:30.929 INFO:teuthology.orchestra.run.smithi152.stdout:18.0.0-7537.gbcbf278c.el9 2023-12-03T11:41:30.929 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7537.gbcbf278c.el9 2023-12-03T11:41:30.930 INFO:teuthology.task.install:The correct ceph version 18.0.0-7537.gbcbf278c is installed. 2023-12-03T11:41:30.930 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-03T11:41:30.930 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:30.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-03T11:41:30.960 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:30.960 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-03T11:41:31.000 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-03T11:41:31.000 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:31.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2023-12-03T11:41:31.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-03T11:41:31.095 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:31.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2023-12-03T11:41:31.122 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-03T11:41:31.189 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-03T11:41:31.190 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:31.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-03T11:41:31.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-03T11:41:31.285 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:31.285 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-03T11:41:31.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-03T11:41:31.379 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-03T11:41:31.380 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:31.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2023-12-03T11:41:31.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-03T11:41:31.473 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:31.473 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2023-12-03T11:41:31.500 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-03T11:41:31.567 INFO:teuthology.run_tasks:Running task ceph... 2023-12-03T11:41:31.640 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-03T11:41:31.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2023-12-03T11:41:31.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2023-12-03T11:41:31.671 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-03T11:41:31.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-03T11:41:31.712 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-03T11:41:31.738 INFO:tasks.ceph:Creating extra log directories... 2023-12-03T11:41:31.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-03T11:41:31.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-03T11:41:31.817 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-03T11:41:31.817 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-03T11:41:31.817 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150', 'branch': 'wip-alimasa-multi-pubsub-etag', 'description': 'rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7476150', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-alimasa-multi-pubsub-etag'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, 'openssl_keys': {'rgw.client.0': {'ca': 'root', 'client': 'client.0', 'embed-key': True}, 'root': {'client': 'client.0', 'cn': 'teuthology', 'install': ['client.0'], 'key-type': 'rsa:4096'}}, 'rgw': {'client.0': {'ssl certificate': 'rgw.client.0'}, 'compression type': 'random', 'ec-data-pool': True, 'frontend': 'beast', 'inline data': False, 'storage classes': 'LUKEWARM, FROZEN'}, 's3tests': {'force-branch': 'ceph-master', 'slow_backend': True, 'storage classes': 'LUKEWARM, FROZEN'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-alimasa-multi-pubsub-etag', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, 'owner': 'scheduled_alimasa@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-alimasa-multi-pubsub-etag', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWklY4I2diZT214tI3xTz6GRNhwlcyPni6UlOvuW/o+2IoxhJOzik5ptxGmaxuz2N7CpKMa/Odz5B5mVb3znSg=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsJ+OYRQMdJGRs6a9/kmeo3WSNgQsUqBVDEgU4jFR4A5K8Xqv6bW0WMEOXZjF3goVFqedsSiXKs+J2O59oiA7w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'tox': ['client.0']}, {'workunit': {'clients': {'client.0': ['rgw/run-bucket-check.sh']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh']}}}, {'workunit': {'clients': {'client.0': ['rgw/test_rgw_s3_mp_reupload.sh']}}}, {'ragweed': {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-reshard.sh']}}}, {'s3tests-java': {'client.0': {'force-branch': 'ceph-master', 'force-repo': 'https://github.com/ceph/java_s3tests.git'}}}, {'s3tests': {'client.0': {'rgw_server': 'client.0'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-versioning.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-03_11:04:43', 'tube': 'smithi', 'user': 'alimasa', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-03T11:41:31.818 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-03T11:41:31.846 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-03T11:41:31.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2023-12-03T11:41:31.900 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2023-12-03T11:41:31.928 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:31.928 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2023-12-03T11:41:31.980 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-03T11:41:31.981 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 658 Links: 1 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-12-03 11:41:24.107038801 +0000 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-12-03 11:39:35.987409577 +0000 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-12-03 11:39:35.987409577 +0000 2023-12-03T11:41:32.035 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-12-03 11:39:35.987409577 +0000 2023-12-03T11:41:32.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-03T11:41:32.100 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-12-03T11:41:32.100 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-12-03T11:41:32.100 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000222043 s, 2.3 MB/s 2023-12-03T11:41:32.101 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-03T11:41:32.157 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 679 Links: 1 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-12-03 11:41:24.107038801 +0000 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-12-03 11:39:36.485401429 +0000 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-12-03 11:39:36.485401429 +0000 2023-12-03T11:41:32.212 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-12-03 11:39:36.485401429 +0000 2023-12-03T11:41:32.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-03T11:41:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-12-03T11:41:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-12-03T11:41:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000204167 s, 2.5 MB/s 2023-12-03T11:41:32.279 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-03T11:41:32.334 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2023-12-03T11:41:32.390 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-03T11:41:32.390 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.390 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 697 Links: 1 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-12-03 11:41:24.107038801 +0000 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-12-03 11:39:36.969393509 +0000 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-12-03 11:39:36.969393509 +0000 2023-12-03T11:41:32.391 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-12-03 11:39:36.969393509 +0000 2023-12-03T11:41:32.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-03T11:41:32.456 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-12-03T11:41:32.456 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-12-03T11:41:32.456 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000190227 s, 2.7 MB/s 2023-12-03T11:41:32.457 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-03T11:41:32.513 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 716 Links: 1 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-12-03 11:41:24.108038785 +0000 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-12-03 11:39:37.449385654 +0000 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-12-03 11:39:37.449385654 +0000 2023-12-03T11:41:32.568 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-12-03 11:39:37.449385654 +0000 2023-12-03T11:41:32.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-03T11:41:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-12-03T11:41:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-12-03T11:41:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000193315 s, 2.6 MB/s 2023-12-03T11:41:32.636 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-03T11:41:32.692 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:32.692 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:32.692 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2023-12-03T11:41:32.709 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-03T11:41:32.709 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2023-12-03 11:41:23.853593366 +0000 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2023-12-03 11:39:35.989639822 +0000 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2023-12-03 11:39:35.989639822 +0000 2023-12-03T11:41:32.766 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2023-12-03 11:39:35.989639822 +0000 2023-12-03T11:41:32.766 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-03T11:41:32.834 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2023-12-03T11:41:32.835 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2023-12-03T11:41:32.835 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000209319 s, 2.4 MB/s 2023-12-03T11:41:32.836 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-03T11:41:32.894 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2023-12-03T11:41:32.951 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-03T11:41:32.951 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:32.951 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2023-12-03 11:41:23.853593366 +0000 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2023-12-03 11:39:36.494633818 +0000 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2023-12-03 11:39:36.494633818 +0000 2023-12-03T11:41:32.952 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2023-12-03 11:39:36.494633818 +0000 2023-12-03T11:41:32.952 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-03T11:41:33.018 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2023-12-03T11:41:33.019 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2023-12-03T11:41:33.019 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000189759 s, 2.7 MB/s 2023-12-03T11:41:33.020 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-03T11:41:33.077 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2023-12-03T11:41:33.133 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-03T11:41:33.133 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2023-12-03 11:41:23.854593354 +0000 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2023-12-03 11:39:36.982628016 +0000 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2023-12-03 11:39:36.982628016 +0000 2023-12-03T11:41:33.134 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2023-12-03 11:39:36.982628016 +0000 2023-12-03T11:41:33.134 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-03T11:41:33.200 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2023-12-03T11:41:33.200 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2023-12-03T11:41:33.201 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000211598 s, 2.4 MB/s 2023-12-03T11:41:33.201 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-03T11:41:33.259 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2023-12-03T11:41:33.314 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2023-12-03 11:41:23.854593354 +0000 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2023-12-03 11:39:37.467622250 +0000 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2023-12-03 11:39:37.467622250 +0000 2023-12-03T11:41:33.315 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2023-12-03 11:39:37.467622250 +0000 2023-12-03T11:41:33.315 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-03T11:41:33.381 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2023-12-03T11:41:33.381 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2023-12-03T11:41:33.381 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000172429 s, 3.0 MB/s 2023-12-03T11:41:33.382 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-03T11:41:33.438 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:33.438 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-03T11:41:33.438 INFO:tasks.ceph:Generating config... 2023-12-03T11:41:33.439 INFO:tasks.ceph:[client] debug rgw = 20 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw bucket counters cache = True 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw max chunk size = 4194304 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw obj stripe size = 6291456 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw torrent flag = True 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] rgw user counters cache = True 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] setgroup = ceph 2023-12-03T11:41:33.440 INFO:tasks.ceph:[client] setuser = ceph 2023-12-03T11:41:33.440 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-03T11:41:33.440 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-03T11:41:33.440 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2023-12-03T11:41:33.440 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-03T11:41:33.441 INFO:tasks.ceph:[mon] lockdep = True 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-03T11:41:33.441 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] lockdep = True 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-03T11:41:33.442 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-03T11:41:33.442 INFO:tasks.ceph:Setting up mon.a... 2023-12-03T11:41:33.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-03T11:41:33.496 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2023-12-03T11:41:33.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-03T11:41:33.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:41:33.570 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116'), ('mon.c', '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]'), ('mon.b', '172.21.15.152')] 2023-12-03T11:41:33.570 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.152', 'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'lockdep': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-03T11:41:33.571 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:33.571 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:41:33.625 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --addv c '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]' --add b 172.21.15.152 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:41:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:41:33.709 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = pacific 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 16 (pacific) 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:41:33.710 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-03T11:41:33.712 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:41:33.766 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1fb29ff4-87be-4f29-84b9-4dc87a2589ed... 2023-12-03T11:41:33.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-03T11:41:33.809 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.856 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-03T11:41:33.858 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.152" 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 5000 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_pg_log_entries = 10 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: osd_min_pg_log_entries = 10 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.859 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2023-12-03T11:41:33.860 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2023-12-03T11:41:33.861 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2023-12-03T11:41:33.862 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: lockdep = True 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: osd shutdown pgref assert = True 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-12-03T11:41:33.863 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2023-12-03T11:41:33.864 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: lockdep = True 2023-12-03T11:41:33.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: debug rgw = 20 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw bucket counters cache = True 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt require ssl = False 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms backend = testing 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw max chunk size = 4194304 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw obj stripe size = 6291456 2023-12-03T11:41:33.866 INFO:teuthology.orchestra.run.smithi116.stdout: rgw torrent flag = True 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout: rgw user counters cache = True 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout: setgroup = ceph 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout: setuser = ceph 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2023-12-03T11:41:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2023-12-03T11:41:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2023-12-03T11:41:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = "172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.152" 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 5000 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: osd_max_pg_log_entries = 10 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: osd_min_pg_log_entries = 10 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2023-12-03T11:41:33.871 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2023-12-03T11:41:33.872 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: bdev async discard = True 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: bdev enable discard = True 2023-12-03T11:41:33.873 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore allocator = bitmap 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 1/20 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 1/20 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 4/10 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: lockdep = True 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2023-12-03T11:41:33.874 INFO:teuthology.orchestra.run.smithi152.stdout: osd shutdown pgref assert = True 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2023-12-03T11:41:33.875 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-03T11:41:33.876 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: lockdep = True 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-03T11:41:33.877 INFO:teuthology.orchestra.run.smithi152.stdout: debug rgw = 20 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw bucket counters cache = True 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw crypt require ssl = False 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw crypt s3 kms backend = testing 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw max chunk size = 4194304 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw obj stripe size = 6291456 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw torrent flag = True 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: rgw user counters cache = True 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: setgroup = ceph 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout: setuser = ceph 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.c] 2023-12-03T11:41:33.878 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.b] 2023-12-03T11:41:33.880 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-03T11:41:33.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-03T11:41:33.976 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-03T11:41:33.976 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:33.976 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-03T11:41:33.991 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:33.991 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-03T11:41:34.045 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:41:34.046 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:34.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-03T11:41:34.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:41:34.124 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:41:34.125 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:34.179 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:41:34.179 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:34.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-03T11:41:34.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:41:34.218 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:41:34.219 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:34.273 INFO:tasks.ceph:Setting up mon nodes... 2023-12-03T11:41:34.273 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-03T11:41:34.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-03T11:41:34.327 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-03T11:41:34.330 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:41:34.402 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:41:34.404 INFO:tasks.ceph:Setting up mds nodes... 2023-12-03T11:41:34.404 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-03T11:41:34.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-03T11:41:34.446 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-03T11:41:34.462 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-03T11:41:34.504 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-03T11:41:34.519 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-03T11:41:34.519 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-03T11:41:34.519 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-03T11:41:34.546 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:34.546 INFO:tasks.ceph:role: osd.0 2023-12-03T11:41:34.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:41:34.546 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:34.621 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:35.126 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-12-03T11:41:35.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-12-03T11:41:35.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-03T11:41:35.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-03T11:41:35.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-03T11:41:35.353 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:35.353 INFO:tasks.ceph:role: osd.1 2023-12-03T11:41:35.353 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:41:35.353 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:35.429 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:35.892 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-12-03T11:41:35.894 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-12-03T11:41:35.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-03T11:41:35.988 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-03T11:41:36.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-03T11:41:36.091 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:36.091 INFO:tasks.ceph:role: osd.2 2023-12-03T11:41:36.091 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:41:36.091 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:36.650 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-12-03T11:41:36.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-12-03T11:41:36.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-03T11:41:36.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-03T11:41:36.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-03T11:41:36.919 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:36.919 INFO:tasks.ceph:role: osd.3 2023-12-03T11:41:36.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:41:36.919 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-03T11:41:36.995 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:36.996 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:37.510 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-12-03T11:41:37.512 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-12-03T11:41:37.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-03T11:41:37.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-03T11:41:37.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:37.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.765+0000 7f23e6a0f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-03T11:41:37.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.766+0000 7f23e6a0f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-03T11:41:37.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.766+0000 7f23e6a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:41:37.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.766+0000 7f23e6a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:41:37.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.766+0000 7f23e6a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:41:37.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:37.767+0000 7f23e6a0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-03T11:41:40.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-03T11:41:40.644 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.733+0000 7f6154d15500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-03T11:41:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.734+0000 7f6154d15500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-03T11:41:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.734+0000 7f6154d15500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:41:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.734+0000 7f6154d15500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:41:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.734+0000 7f6154d15500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:41:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:40.734+0000 7f6154d15500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-03T11:41:43.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-03T11:41:43.616 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.707+0000 7f93ac4cf500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-03T11:41:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.708+0000 7f93ac4cf500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-03T11:41:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.708+0000 7f93ac4cf500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:41:43.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.708+0000 7f93ac4cf500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:41:43.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.708+0000 7f93ac4cf500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:41:43.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:43.708+0000 7f93ac4cf500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-03T11:41:46.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-03T11:41:46.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.679+0000 7fe3e1e0c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.679+0000 7fe3e1e0c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.679+0000 7fe3e1e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.679+0000 7fe3e1e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.679+0000 7fe3e1e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:41:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:41:46.680+0000 7fe3e1e0c500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-03T11:41:49.529 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-03T11:41:49.561 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-03T11:41:49.592 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:49.592 INFO:tasks.ceph:role: osd.4 2023-12-03T11:41:49.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:41:49.593 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:49.670 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:49.671 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:49.671 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:50.191 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2023-12-03T11:41:50.193 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2023-12-03T11:41:50.193 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-03T11:41:50.336 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-03T11:41:50.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-03T11:41:50.444 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:50.444 INFO:tasks.ceph:role: osd.5 2023-12-03T11:41:50.444 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:41:50.444 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:50.520 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:50.521 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:50.521 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:50.521 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:50.972 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2023-12-03T11:41:50.974 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2023-12-03T11:41:50.974 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-03T11:41:51.061 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-03T11:41:51.094 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-03T11:41:51.162 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:51.162 INFO:tasks.ceph:role: osd.6 2023-12-03T11:41:51.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:41:51.163 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-03T11:41:51.239 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:51.240 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:51.737 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2023-12-03T11:41:51.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2023-12-03T11:41:51.740 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-03T11:41:51.866 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-03T11:41:51.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-03T11:41:51.968 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:41:51.968 INFO:tasks.ceph:role: osd.7 2023-12-03T11:41:51.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:41:51.969 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-03T11:41:52.042 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:41:52.042 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:41:52.042 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:41:52.042 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:41:52.043 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:41:52.570 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2023-12-03T11:41:52.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2023-12-03T11:41:52.572 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-03T11:41:52.700 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-03T11:41:52.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:52.831 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.829+0000 7f3182025500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-03T11:41:52.831 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.830+0000 7f3182025500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-03T11:41:52.832 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.830+0000 7f3182025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:41:52.832 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.830+0000 7f3182025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:41:52.832 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.830+0000 7f3182025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:41:52.832 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:52.830+0000 7f3182025500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-03T11:41:55.687 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-03T11:41:55.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:55.809 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.807+0000 7fb04b867500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-03T11:41:55.809 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.807+0000 7fb04b867500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-03T11:41:55.809 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.807+0000 7fb04b867500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:41:55.809 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.807+0000 7fb04b867500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:41:55.810 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.807+0000 7fb04b867500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:41:55.810 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:55.808+0000 7fb04b867500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-03T11:41:58.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-03T11:41:58.678 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.764+0000 7f073b6d4500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.764+0000 7f073b6d4500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.764+0000 7f073b6d4500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.764+0000 7f073b6d4500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.764+0000 7f073b6d4500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:41:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:41:58.765+0000 7f073b6d4500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-03T11:42:01.624 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-03T11:42:01.655 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:42:01.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.743+0000 7f5c0ac1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-03T11:42:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.743+0000 7f5c0ac1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-03T11:42:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.743+0000 7f5c0ac1a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:42:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.743+0000 7f5c0ac1a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:42:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.743+0000 7f5c0ac1a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:42:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-03T11:42:01.744+0000 7f5c0ac1a500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-03T11:42:04.591 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-03T11:42:04.622 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-03T11:42:04.623 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:04.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-03T11:42:04.655 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:04.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-03T11:42:04.726 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:04.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-03T11:42:04.796 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:04.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-03T11:42:04.865 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:04.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-03T11:42:04.937 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:04.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-03T11:42:04.969 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:04.969 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-03T11:42:05.039 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:05.040 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-03T11:42:05.110 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:05.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-03T11:42:05.179 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:05.180 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-03T11:42:05.250 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:05.250 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-03T11:42:05.265 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:05.265 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-03T11:42:05.307 INFO:tasks.ceph:Adding keys to all mons... 2023-12-03T11:42:05.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-03T11:42:05.309 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBuaWxleRSCExAA+kUIFD9NPP/Z4PDt2xBkDg== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBxaWxleAa0LRAAzYrHbwokTITRj576abYOWA== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB0aWxl4F/LKxAAV1k3l9J4NlGphLphKfVP2Q== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB3aWxlsAtBKhAAXEed4b26TMlP8VlFYF2daA== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB6aWxl4hKNKBAA+PdVibN3MB7O3uj1sR1LDw== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBuaWxlCNHxFxAALiKCiGxgwtIVSNFk4G40Yw== 2023-12-03T11:42:05.351 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCAaWxl4vSFMRAA/RboBT7PqXZtHGNjzMinSQ== 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCDaWxl3xAyMBAAbz8Z885X7xPGW4OtKbnZ7w== 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCGaWxlhVueLRAAS6Tur0Cv8e4IOpJcun4K8A== 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCJaWxl26xiLBAAW/dMPcghcRLmwUhV3t46Yw== 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBuaWxligacGhAA7+nkoGrCbUplHfTTLqFeKQ== 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2023-12-03T11:42:05.352 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBuaWxl8XAJHhAAKq+EN9bmEfVL8l7bJpaFtw== 2023-12-03T11:42:05.374 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.y] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBuaWxleRSCExAA+kUIFD9NPP/Z4PDt2xBkDg== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBxaWxleAa0LRAAzYrHbwokTITRj576abYOWA== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQB0aWxl4F/LKxAAV1k3l9J4NlGphLphKfVP2Q== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQB3aWxlsAtBKhAAXEed4b26TMlP8VlFYF2daA== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQB6aWxl4hKNKBAA+PdVibN3MB7O3uj1sR1LDw== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBuaWxlCNHxFxAALiKCiGxgwtIVSNFk4G40Yw== 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.4] 2023-12-03T11:42:05.375 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCAaWxl4vSFMRAA/RboBT7PqXZtHGNjzMinSQ== 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.5] 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCDaWxl3xAyMBAAbz8Z885X7xPGW4OtKbnZ7w== 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.6] 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCGaWxlhVueLRAAS6Tur0Cv8e4IOpJcun4K8A== 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.7] 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCJaWxl26xiLBAAW/dMPcghcRLmwUhV3t46Yw== 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBuaWxligacGhAA7+nkoGrCbUplHfTTLqFeKQ== 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout:[client.1] 2023-12-03T11:42:05.376 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQBuaWxl8XAJHhAAKq+EN9bmEfVL8l7bJpaFtw== 2023-12-03T11:42:05.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:42:05.396 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:42:05.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.474 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.528 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.582 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.636 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:42:05.689 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:42:05.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.743 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.849 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.903 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:42:05.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:42:05.957 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:42:06.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:42:06.013 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:42:06.067 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-03T11:42:06.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-03T11:42:06.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:42:06.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-03T11:42:06.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-03T11:42:06.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:42:06.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-03T11:42:06.408 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-03T11:42:06.435 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:42:06.531 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-03T11:42:06.558 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:42:06.561 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:42:06.613 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-03T11:42:06.613 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-03T11:42:06.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-03T11:42:06.615 INFO:tasks.ceph.mon.a:Started 2023-12-03T11:42:06.615 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-03T11:42:06.616 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-03T11:42:06.617 INFO:tasks.ceph.mon.c:Started 2023-12-03T11:42:06.617 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-03T11:42:06.617 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-03T11:42:06.655 INFO:tasks.ceph.mon.b:Started 2023-12-03T11:42:06.656 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-03T11:42:06.656 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-03T11:42:06.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-03T11:42:06.658 INFO:tasks.ceph.mgr.y:Started 2023-12-03T11:42:06.658 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-03T11:42:06.658 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-03T11:42:06.660 INFO:tasks.ceph.mgr.x:Started 2023-12-03T11:42:06.660 INFO:tasks.ceph:Setting crush tunables to default 2023-12-03T11:42:06.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2023-12-03T11:42:12.453 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:12.453+0000 7fa5ed82e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-03T11:42:12.504 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:12.503+0000 7fa5ed82e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-03T11:42:12.839 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2023-12-03T11:42:12.851 INFO:tasks.ceph:check_enable_crimson: False 2023-12-03T11:42:12.851 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-03T11:42:12.851 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:12.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-03T11:42:12.879 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:12.879 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-03T11:42:12.946 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:12.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-03T11:42:13.014 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:13.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-03T11:42:13.038 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.038+0000 7fa5ed82e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-03T11:42:13.081 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:13.081 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-03T11:42:13.092 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.091+0000 7fa5ed82e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-03T11:42:13.108 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:13.108 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-03T11:42:13.139 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.138+0000 7fa5ed82e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-03T11:42:13.165 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:13.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-03T11:42:13.195 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.194+0000 7fa5ed82e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-03T11:42:13.221 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-03T11:42:13.221 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-03T11:42:13.239 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.239+0000 7fa5ed82e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-03T11:42:13.265 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new d749d2c6-759f-4813-9c65-e6e961223faa 0 2023-12-03T11:42:13.346 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.345+0000 7fa5ed82e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-03T11:42:13.420 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.419+0000 7fa5ed82e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-03T11:42:13.467 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.466+0000 7fa5ed82e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-03T11:42:13.494 INFO:teuthology.orchestra.run.smithi152.stdout:0 2023-12-03T11:42:13.500 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:13.499+0000 7fd85e456040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-03T11:42:13.505 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 42944e52-7f23-4124-8379-9ac906ffc913 1 2023-12-03T11:42:13.522 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.522+0000 7fa5ed82e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-03T11:42:13.552 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:13.550+0000 7fd85e456040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-03T11:42:13.572 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.571+0000 7fa5ed82e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-03T11:42:13.710 INFO:teuthology.orchestra.run.smithi152.stdout:1 2023-12-03T11:42:13.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new f065550a-e864-4719-90ee-bcabe5ee0d5c 2 2023-12-03T11:42:13.911 INFO:teuthology.orchestra.run.smithi152.stdout:2 2023-12-03T11:42:13.923 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 9e4b08fe-f648-4a54-a999-50d71b3a8972 3 2023-12-03T11:42:13.971 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:13.970+0000 7fa5ed82e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-03T11:42:14.101 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.100+0000 7fd85e456040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-03T11:42:14.111 INFO:teuthology.orchestra.run.smithi152.stdout:3 2023-12-03T11:42:14.112 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.111+0000 7fa5ed82e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-03T11:42:14.122 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 215f1242-25ac-4985-b735-33a45baa43d9 4 2023-12-03T11:42:14.153 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.152+0000 7fd85e456040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-03T11:42:14.197 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.195+0000 7fd85e456040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-03T11:42:14.235 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.233+0000 7fa5ed82e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-03T11:42:14.250 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.248+0000 7fd85e456040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-03T11:42:14.282 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.280+0000 7fa5ed82e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-03T11:42:14.291 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.289+0000 7fd85e456040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-03T11:42:14.310 INFO:teuthology.orchestra.run.smithi152.stdout:4 2023-12-03T11:42:14.321 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 9351f254-5e3f-4b32-b51f-d5d596978130 5 2023-12-03T11:42:14.385 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.384+0000 7fa5ed82e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:42:14.391 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.389+0000 7fd85e456040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-03T11:42:14.460 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.459+0000 7fd85e456040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-03T11:42:14.483 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.482+0000 7fa5ed82e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-03T11:42:14.504 INFO:teuthology.orchestra.run.smithi152.stdout:5 2023-12-03T11:42:14.506 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.504+0000 7fd85e456040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-03T11:42:14.515 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new e174e707-159f-4d8d-8507-1285c79862ca 6 2023-12-03T11:42:14.557 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.556+0000 7fd85e456040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-03T11:42:14.603 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:14.601+0000 7fd85e456040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-03T11:42:14.632 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.630+0000 7fa5ed82e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:42:14.676 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.674+0000 7fa5ed82e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-03T11:42:14.699 INFO:teuthology.orchestra.run.smithi152.stdout:6 2023-12-03T11:42:14.710 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154 7 2023-12-03T11:42:14.823 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:14.822+0000 7fa5ed82e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-03T11:42:14.891 INFO:teuthology.orchestra.run.smithi152.stdout:7 2023-12-03T11:42:14.902 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-03T11:42:14.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-03T11:42:14.904 INFO:tasks.ceph.osd.0:Started 2023-12-03T11:42:14.905 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-03T11:42:14.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-03T11:42:14.906 INFO:tasks.ceph.osd.1:Started 2023-12-03T11:42:14.906 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-03T11:42:14.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-03T11:42:14.908 INFO:tasks.ceph.osd.2:Started 2023-12-03T11:42:14.908 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-03T11:42:14.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-03T11:42:14.910 INFO:tasks.ceph.osd.3:Started 2023-12-03T11:42:14.911 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-03T11:42:14.911 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-03T11:42:14.913 INFO:tasks.ceph.osd.4:Started 2023-12-03T11:42:14.913 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-03T11:42:14.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-03T11:42:14.916 INFO:tasks.ceph.osd.5:Started 2023-12-03T11:42:14.916 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-03T11:42:14.916 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-03T11:42:14.918 INFO:tasks.ceph.osd.6:Started 2023-12-03T11:42:14.918 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-03T11:42:14.918 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-03T11:42:14.920 INFO:tasks.ceph.osd.7:Started 2023-12-03T11:42:14.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:15.016 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.015+0000 7fd85e456040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-03T11:42:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:15.117 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:14.889711+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:15.128 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-03T11:42:15.128 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-03T11:42:15.157 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.155+0000 7fd85e456040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-03T11:42:15.269 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.268+0000 7fd85e456040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-03T11:42:15.315 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.314+0000 7fd85e456040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-03T11:42:15.411 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.410+0000 7fd85e456040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:42:15.505 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.503+0000 7fd85e456040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-03T11:42:15.514 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T11:42:15.513+0000 7efea8c12500 -1 Falling back to public interface 2023-12-03T11:42:15.517 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T11:42:15.515+0000 7f19193f9500 -1 Falling back to public interface 2023-12-03T11:42:15.532 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T11:42:15.531+0000 7f27a5018500 -1 Falling back to public interface 2023-12-03T11:42:15.532 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T11:42:15.531+0000 7f6ab4619500 -1 Falling back to public interface 2023-12-03T11:42:15.535 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T11:42:15.533+0000 7f2d7220c500 -1 Falling back to public interface 2023-12-03T11:42:15.536 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T11:42:15.535+0000 7f2a21d1a500 -1 Falling back to public interface 2023-12-03T11:42:15.546 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T11:42:15.544+0000 7ff22ce0f500 -1 Falling back to public interface 2023-12-03T11:42:15.546 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T11:42:15.545+0000 7ff2f42d4500 -1 Falling back to public interface 2023-12-03T11:42:15.548 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T11:42:15.546+0000 7ff2f327b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6825/0: (98) Address already in use 2023-12-03T11:42:15.654 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.653+0000 7fd85e456040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:42:15.676 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:15.674+0000 7fa5ed82e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-03T11:42:15.696 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.695+0000 7fd85e456040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-03T11:42:15.838 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:15.836+0000 7fd85e456040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-03T11:42:15.933 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:15.947 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:15.946+0000 7fa5ed82e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-03T11:42:15.998 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:15.997+0000 7fa5ed82e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-03T11:42:16.093 INFO:teuthology.misc.health.smithi116.stdout: 2023-12-03T11:42:16.094 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":10,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:14.889711+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:16.101 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:16.099+0000 7fa5ed82e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-03T11:42:16.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-03T11:42:16.663 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:16.659+0000 7fd85e456040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-03T11:42:16.934 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:16.932+0000 7fd85e456040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-03T11:42:16.984 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:16.982+0000 7fd85e456040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-03T11:42:17.085 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-12-03T11:42:17.084+0000 7fd85e456040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-03T11:42:17.126 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:17.125+0000 7fa58e46e640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-03T11:42:17.127 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T11:42:17.125+0000 7fa58e46e640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-03T11:42:18.991 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T11:42:18.990+0000 7efea8c12500 -1 osd.1 0 log_to_monitors true 2023-12-03T11:42:18.996 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T11:42:18.995+0000 7f19193f9500 -1 osd.5 0 log_to_monitors true 2023-12-03T11:42:19.023 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T11:42:19.021+0000 7f2d7220c500 -1 osd.6 0 log_to_monitors true 2023-12-03T11:42:19.037 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T11:42:19.036+0000 7f2a21d1a500 -1 osd.2 0 log_to_monitors true 2023-12-03T11:42:19.056 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T11:42:19.055+0000 7ff22ce0f500 -1 osd.7 0 log_to_monitors true 2023-12-03T11:42:19.067 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T11:42:19.066+0000 7f6ab4619500 -1 osd.3 0 log_to_monitors true 2023-12-03T11:42:19.072 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T11:42:19.070+0000 7ff2f42d4500 -1 osd.4 0 log_to_monitors true 2023-12-03T11:42:19.081 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T11:42:19.080+0000 7f27a5018500 -1 osd.0 0 log_to_monitors true 2023-12-03T11:42:22.909 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:22.937 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T11:42:22.936+0000 7f1914b64640 -1 osd.5 0 waiting for initial osdmap 2023-12-03T11:42:22.950 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T11:42:22.949+0000 7f190d19c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.050 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T11:42:23.048+0000 7f6aaf75b640 -1 osd.3 0 waiting for initial osdmap 2023-12-03T11:42:23.065 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T11:42:23.064+0000 7f6aa7d93640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.069 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T11:42:23.068+0000 7f2a1d485640 -1 osd.2 0 waiting for initial osdmap 2023-12-03T11:42:23.083 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T11:42:23.081+0000 7efea3daa640 -1 osd.1 0 waiting for initial osdmap 2023-12-03T11:42:23.085 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T11:42:23.084+0000 7f2a15abd640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.089 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T11:42:23.087+0000 7ff2efa3f640 -1 osd.4 0 waiting for initial osdmap 2023-12-03T11:42:23.099 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T11:42:23.098+0000 7f2d6d977640 -1 osd.6 0 waiting for initial osdmap 2023-12-03T11:42:23.101 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T11:42:23.100+0000 7efe9c3e2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.104 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T11:42:23.103+0000 7ff2e8077640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.107 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T11:42:23.106+0000 7f2d65faf640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.109 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T11:42:23.107+0000 7f27a00c3640 -1 osd.0 0 waiting for initial osdmap 2023-12-03T11:42:23.120 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T11:42:23.119+0000 7f27986fb640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.124 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T11:42:23.123+0000 7ff227eab640 -1 osd.7 0 waiting for initial osdmap 2023-12-03T11:42:23.141 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T11:42:23.139+0000 7ff2204e3640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:42:23.252 INFO:teuthology.misc.health.smithi116.stdout: 2023-12-03T11:42:23.252 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":12,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:20.993136+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:23.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-03T11:42:30.073 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:30.354 INFO:teuthology.misc.health.smithi116.stdout: 2023-12-03T11:42:30.354 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":16,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:26.962299+0000","last_up_change":"2023-12-03T11:42:23.942511+0000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:42:24.163469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6809","nonce":1519417773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6811","nonce":1519417773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6820","nonce":1519417773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6816","nonce":1519417773}]},"public_addr":"172.21.15.116:6809/1519417773","cluster_addr":"172.21.15.116:6811/1519417773","heartbeat_back_addr":"172.21.15.116:6820/1519417773","heartbeat_front_addr":"172.21.15.116:6816/1519417773","state":["exists","up"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6801","nonce":3848060595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6803","nonce":3848060595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6807","nonce":3848060595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6805","nonce":3848060595}]},"public_addr":"172.21.15.116:6801/3848060595","cluster_addr":"172.21.15.116:6803/3848060595","heartbeat_back_addr":"172.21.15.116:6807/3848060595","heartbeat_front_addr":"172.21.15.116:6805/3848060595","state":["exists","up"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6825","nonce":4019364149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6827","nonce":4019364149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6831","nonce":4019364149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6829","nonce":4019364149}]},"public_addr":"172.21.15.116:6825/4019364149","cluster_addr":"172.21.15.116:6827/4019364149","heartbeat_back_addr":"172.21.15.116:6831/4019364149","heartbeat_front_addr":"172.21.15.116:6829/4019364149","state":["exists","up"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6813","nonce":178738993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6817","nonce":178738993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6823","nonce":178738993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6821","nonce":178738993}]},"public_addr":"172.21.15.116:6813/178738993","cluster_addr":"172.21.15.116:6817/178738993","heartbeat_back_addr":"172.21.15.116:6823/178738993","heartbeat_front_addr":"172.21.15.116:6821/178738993","state":["exists","up"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6817","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6819","nonce":2654994651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6823","nonce":2654994651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6831","nonce":2654994651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6828","nonce":2654994651}]},"public_addr":"172.21.15.152:6819/2654994651","cluster_addr":"172.21.15.152:6823/2654994651","heartbeat_back_addr":"172.21.15.152:6831/2654994651","heartbeat_front_addr":"172.21.15.152:6828/2654994651","state":["exists","up"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6801","nonce":1890087800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6803","nonce":1890087800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6807","nonce":1890087800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6805","nonce":1890087800}]},"public_addr":"172.21.15.152:6801/1890087800","cluster_addr":"172.21.15.152:6803/1890087800","heartbeat_back_addr":"172.21.15.152:6807/1890087800","heartbeat_front_addr":"172.21.15.152:6805/1890087800","state":["exists","up"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6809","nonce":3202692571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6811","nonce":3202692571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6815","nonce":3202692571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6813","nonce":3202692571}]},"public_addr":"172.21.15.152:6809/3202692571","cluster_addr":"172.21.15.152:6811/3202692571","heartbeat_back_addr":"172.21.15.152:6815/3202692571","heartbeat_front_addr":"172.21.15.152:6813/3202692571","state":["exists","up"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6818","nonce":1133785446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6822","nonce":1133785446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6827","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6829","nonce":1133785446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6825","nonce":1133785446}]},"public_addr":"172.21.15.152:6818/1133785446","cluster_addr":"172.21.15.152:6822/1133785446","heartbeat_back_addr":"172.21.15.152:6829/1133785446","heartbeat_front_addr":"172.21.15.152:6825/1133785446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.057388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.000120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.060796+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.101935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.069975+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.010407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.048462+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.023413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:30.365 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-03T11:42:30.365 INFO:tasks.ceph:Creating RBD pool 2023-12-03T11:42:30.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-03T11:42:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2023-12-03T11:42:30.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-03T11:42:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-03T11:42:32.000 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-03T11:42:32.000 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-03T11:42:32.001 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-03T11:42:32.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-03T11:42:32.001 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-03T11:42:32.331 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:32.342 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2965493816},{"type":"v1","addr":"172.21.15.152:6833","nonce":2965493816}]},"active_addr":"172.21.15.152:6833/2965493816","active_change":"2023-12-03T11:42:16.114540+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":1056901091}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":1582688130}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":4019099594}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":1735960631}]}]} 2023-12-03T11:42:32.350 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-03T11:42:32.350 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-03T11:42:32.350 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:32.628 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:32.628 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:31.977133+0000","last_up_change":"2023-12-03T11:42:23.942511+0000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:42:24.163469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:42:30.657321+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6809","nonce":1519417773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6811","nonce":1519417773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6820","nonce":1519417773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6816","nonce":1519417773}]},"public_addr":"172.21.15.116:6809/1519417773","cluster_addr":"172.21.15.116:6811/1519417773","heartbeat_back_addr":"172.21.15.116:6820/1519417773","heartbeat_front_addr":"172.21.15.116:6816/1519417773","state":["exists","up"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6801","nonce":3848060595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6803","nonce":3848060595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6807","nonce":3848060595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6805","nonce":3848060595}]},"public_addr":"172.21.15.116:6801/3848060595","cluster_addr":"172.21.15.116:6803/3848060595","heartbeat_back_addr":"172.21.15.116:6807/3848060595","heartbeat_front_addr":"172.21.15.116:6805/3848060595","state":["exists","up"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6825","nonce":4019364149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6827","nonce":4019364149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6831","nonce":4019364149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6829","nonce":4019364149}]},"public_addr":"172.21.15.116:6825/4019364149","cluster_addr":"172.21.15.116:6827/4019364149","heartbeat_back_addr":"172.21.15.116:6831/4019364149","heartbeat_front_addr":"172.21.15.116:6829/4019364149","state":["exists","up"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6813","nonce":178738993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6817","nonce":178738993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6823","nonce":178738993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6821","nonce":178738993}]},"public_addr":"172.21.15.116:6813/178738993","cluster_addr":"172.21.15.116:6817/178738993","heartbeat_back_addr":"172.21.15.116:6823/178738993","heartbeat_front_addr":"172.21.15.116:6821/178738993","state":["exists","up"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6817","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6819","nonce":2654994651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6823","nonce":2654994651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6831","nonce":2654994651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6828","nonce":2654994651}]},"public_addr":"172.21.15.152:6819/2654994651","cluster_addr":"172.21.15.152:6823/2654994651","heartbeat_back_addr":"172.21.15.152:6831/2654994651","heartbeat_front_addr":"172.21.15.152:6828/2654994651","state":["exists","up"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6801","nonce":1890087800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6803","nonce":1890087800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6807","nonce":1890087800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6805","nonce":1890087800}]},"public_addr":"172.21.15.152:6801/1890087800","cluster_addr":"172.21.15.152:6803/1890087800","heartbeat_back_addr":"172.21.15.152:6807/1890087800","heartbeat_front_addr":"172.21.15.152:6805/1890087800","state":["exists","up"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6809","nonce":3202692571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6811","nonce":3202692571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6815","nonce":3202692571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6813","nonce":3202692571}]},"public_addr":"172.21.15.152:6809/3202692571","cluster_addr":"172.21.15.152:6811/3202692571","heartbeat_back_addr":"172.21.15.152:6815/3202692571","heartbeat_front_addr":"172.21.15.152:6813/3202692571","state":["exists","up"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6818","nonce":1133785446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6822","nonce":1133785446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6827","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6829","nonce":1133785446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6825","nonce":1133785446}]},"public_addr":"172.21.15.152:6818/1133785446","cluster_addr":"172.21.15.152:6822/1133785446","heartbeat_back_addr":"172.21.15.152:6829/1133785446","heartbeat_front_addr":"172.21.15.152:6825/1133785446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.057388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.000120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.060796+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.101935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.069975+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.010407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.048462+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.023413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:32.640 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-03T11:42:32.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:42:32.918 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:32.919 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"fsid":"1fb29ff4-87be-4f29-84b9-4dc87a2589ed","created":"2023-12-03T11:42:11.973556+0000","modified":"2023-12-03T11:42:31.977133+0000","last_up_change":"2023-12-03T11:42:23.942511+0000","last_in_change":"2023-12-03T11:42:14.889711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:42:24.163469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:42:30.657321+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d749d2c6-759f-4813-9c65-e6e961223faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6809","nonce":1519417773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6811","nonce":1519417773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6820","nonce":1519417773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":1519417773},{"type":"v1","addr":"172.21.15.116:6816","nonce":1519417773}]},"public_addr":"172.21.15.116:6809/1519417773","cluster_addr":"172.21.15.116:6811/1519417773","heartbeat_back_addr":"172.21.15.116:6820/1519417773","heartbeat_front_addr":"172.21.15.116:6816/1519417773","state":["exists","up"]},{"osd":1,"uuid":"42944e52-7f23-4124-8379-9ac906ffc913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6801","nonce":3848060595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6803","nonce":3848060595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6807","nonce":3848060595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3848060595},{"type":"v1","addr":"172.21.15.116:6805","nonce":3848060595}]},"public_addr":"172.21.15.116:6801/3848060595","cluster_addr":"172.21.15.116:6803/3848060595","heartbeat_back_addr":"172.21.15.116:6807/3848060595","heartbeat_front_addr":"172.21.15.116:6805/3848060595","state":["exists","up"]},{"osd":2,"uuid":"f065550a-e864-4719-90ee-bcabe5ee0d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6825","nonce":4019364149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6827","nonce":4019364149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6831","nonce":4019364149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":4019364149},{"type":"v1","addr":"172.21.15.116:6829","nonce":4019364149}]},"public_addr":"172.21.15.116:6825/4019364149","cluster_addr":"172.21.15.116:6827/4019364149","heartbeat_back_addr":"172.21.15.116:6831/4019364149","heartbeat_front_addr":"172.21.15.116:6829/4019364149","state":["exists","up"]},{"osd":3,"uuid":"9e4b08fe-f648-4a54-a999-50d71b3a8972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6813","nonce":178738993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6817","nonce":178738993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6823","nonce":178738993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":178738993},{"type":"v1","addr":"172.21.15.116:6821","nonce":178738993}]},"public_addr":"172.21.15.116:6813/178738993","cluster_addr":"172.21.15.116:6817/178738993","heartbeat_back_addr":"172.21.15.116:6823/178738993","heartbeat_front_addr":"172.21.15.116:6821/178738993","state":["exists","up"]},{"osd":4,"uuid":"215f1242-25ac-4985-b735-33a45baa43d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6817","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6819","nonce":2654994651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6823","nonce":2654994651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6831","nonce":2654994651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2654994651},{"type":"v1","addr":"172.21.15.152:6828","nonce":2654994651}]},"public_addr":"172.21.15.152:6819/2654994651","cluster_addr":"172.21.15.152:6823/2654994651","heartbeat_back_addr":"172.21.15.152:6831/2654994651","heartbeat_front_addr":"172.21.15.152:6828/2654994651","state":["exists","up"]},{"osd":5,"uuid":"9351f254-5e3f-4b32-b51f-d5d596978130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6801","nonce":1890087800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6803","nonce":1890087800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6807","nonce":1890087800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":1890087800},{"type":"v1","addr":"172.21.15.152:6805","nonce":1890087800}]},"public_addr":"172.21.15.152:6801/1890087800","cluster_addr":"172.21.15.152:6803/1890087800","heartbeat_back_addr":"172.21.15.152:6807/1890087800","heartbeat_front_addr":"172.21.15.152:6805/1890087800","state":["exists","up"]},{"osd":6,"uuid":"e174e707-159f-4d8d-8507-1285c79862ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6809","nonce":3202692571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6811","nonce":3202692571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6815","nonce":3202692571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3202692571},{"type":"v1","addr":"172.21.15.152:6813","nonce":3202692571}]},"public_addr":"172.21.15.152:6809/3202692571","cluster_addr":"172.21.15.152:6811/3202692571","heartbeat_back_addr":"172.21.15.152:6815/3202692571","heartbeat_front_addr":"172.21.15.152:6813/3202692571","state":["exists","up"]},{"osd":7,"uuid":"9ad30e16-0fa7-4dcf-a7d5-a3ca9bd17154","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6818","nonce":1133785446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6822","nonce":1133785446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6827","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6829","nonce":1133785446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1133785446},{"type":"v1","addr":"172.21.15.152:6825","nonce":1133785446}]},"public_addr":"172.21.15.152:6818/1133785446","cluster_addr":"172.21.15.152:6822/1133785446","heartbeat_back_addr":"172.21.15.152:6829/1133785446","heartbeat_front_addr":"172.21.15.152:6825/1133785446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.057388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.000120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.060796+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.101935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.069975+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.010407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.048462+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:42:20.023413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:42:32.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-03T11:42:32.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-03T11:42:32.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-03T11:42:32.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-03T11:42:32.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-03T11:42:32.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-03T11:42:32.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-03T11:42:32.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-03T11:42:33.167 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:33.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:42:33.176 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:33.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-03T11:42:33.178 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:33.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-03T11:42:33.179 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-03T11:42:33.181 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.182 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-12-03T11:42:33.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:42:33.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-03T11:42:33.183 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-03T11:42:33.187 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-12-03T11:42:33.188 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-03T11:42:33.743 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.757 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.757 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-12-03T11:42:33.758 INFO:teuthology.orchestra.run.smithi116.stdout:55834574851 2023-12-03T11:42:33.764 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2023-12-03T11:42:33.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2023-12-03T11:42:33.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2023-12-03T11:42:33.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2023-12-03T11:42:33.796 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:33.799 INFO:teuthology.orchestra.run.smithi116.stdout:55834574851 2023-12-03T11:42:33.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2023-12-03T11:42:33.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-12-03T11:42:33.821 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:33.829 INFO:teuthology.orchestra.run.smithi116.stdout:55834574851 2023-12-03T11:42:33.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-12-03T11:42:33.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.1 2023-12-03T11:42:34.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:42:34.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-03T11:42:34.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-03T11:42:34.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-03T11:42:34.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-03T11:42:34.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-03T11:42:34.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-03T11:42:34.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:42:35.302 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-12-03T11:42:35.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2023-12-03T11:42:35.320 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.352 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:35.353 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-12-03T11:42:35.373 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-12-03T11:42:35.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.2 2023-12-03T11:42:35.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.7 2023-12-03T11:42:35.374 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.374 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.3 2023-12-03T11:42:35.390 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.397 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:35.403 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-12-03T11:42:35.403 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-12-03T11:42:35.407 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-12-03T11:42:35.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.6 2023-12-03T11:42:35.411 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.420 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2023-12-03T11:42:35.420 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.421 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2023-12-03T11:42:35.421 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.421 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.1 2023-12-03T11:42:35.422 DEBUG:teuthology.parallel:result is None 2023-12-03T11:42:35.422 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-03T11:42:35.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:42:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:35.735 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-12-03T11:42:35.747 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-03T11:42:34.122603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218812,"kb_used_data":2628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755908,"statfs":{"total":773094113280,"available":772870049792,"internally_reserved":0,"allocated":2691072,"data_stored":1667812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000924"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.496368+0000","last_change":"2023-12-03T11:42:31.987483+0000","last_active":"2023-12-03T11:42:33.496368+0000","last_peered":"2023-12-03T11:42:33.496368+0000","last_clean":"2023-12-03T11:42:33.496368+0000","last_became_active":"2023-12-03T11:42:31.987164+0000","last_became_peered":"2023-12-03T11:42:31.987164+0000","last_unstale":"2023-12-03T11:42:33.496368+0000","last_undegraded":"2023-12-03T11:42:33.496368+0000","last_fullsized":"2023-12-03T11:42:33.496368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T15:35:59.006579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.000061+0000","last_change":"2023-12-03T11:42:31.986643+0000","last_active":"2023-12-03T11:42:33.000061+0000","last_peered":"2023-12-03T11:42:33.000061+0000","last_clean":"2023-12-03T11:42:33.000061+0000","last_became_active":"2023-12-03T11:42:31.986407+0000","last_became_peered":"2023-12-03T11:42:31.986407+0000","last_unstale":"2023-12-03T11:42:33.000061+0000","last_undegraded":"2023-12-03T11:42:33.000061+0000","last_fullsized":"2023-12-03T11:42:33.000061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:57:19.678143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497153+0000","last_change":"2023-12-03T11:42:32.001616+0000","last_active":"2023-12-03T11:42:33.497153+0000","last_peered":"2023-12-03T11:42:33.497153+0000","last_clean":"2023-12-03T11:42:33.497153+0000","last_became_active":"2023-12-03T11:42:32.001354+0000","last_became_peered":"2023-12-03T11:42:32.001354+0000","last_unstale":"2023-12-03T11:42:33.497153+0000","last_undegraded":"2023-12-03T11:42:33.497153+0000","last_fullsized":"2023-12-03T11:42:33.497153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:15:21.172972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:32.999951+0000","last_change":"2023-12-03T11:42:32.001658+0000","last_active":"2023-12-03T11:42:32.999951+0000","last_peered":"2023-12-03T11:42:32.999951+0000","last_clean":"2023-12-03T11:42:32.999951+0000","last_became_active":"2023-12-03T11:42:32.001291+0000","last_became_peered":"2023-12-03T11:42:32.001291+0000","last_unstale":"2023-12-03T11:42:32.999951+0000","last_undegraded":"2023-12-03T11:42:32.999951+0000","last_fullsized":"2023-12-03T11:42:32.999951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T20:30:24.051910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497183+0000","last_change":"2023-12-03T11:42:31.985722+0000","last_active":"2023-12-03T11:42:33.497183+0000","last_peered":"2023-12-03T11:42:33.497183+0000","last_clean":"2023-12-03T11:42:33.497183+0000","last_became_active":"2023-12-03T11:42:31.985363+0000","last_became_peered":"2023-12-03T11:42:31.985363+0000","last_unstale":"2023-12-03T11:42:33.497183+0000","last_undegraded":"2023-12-03T11:42:33.497183+0000","last_fullsized":"2023-12-03T11:42:33.497183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:54:56.421628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497176+0000","last_change":"2023-12-03T11:42:32.109247+0000","last_active":"2023-12-03T11:42:33.497176+0000","last_peered":"2023-12-03T11:42:33.497176+0000","last_clean":"2023-12-03T11:42:33.497176+0000","last_became_active":"2023-12-03T11:42:32.108940+0000","last_became_peered":"2023-12-03T11:42:32.108940+0000","last_unstale":"2023-12-03T11:42:33.497176+0000","last_undegraded":"2023-12-03T11:42:33.497176+0000","last_fullsized":"2023-12-03T11:42:33.497176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:23:57.934082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497138+0000","last_change":"2023-12-03T11:42:31.988766+0000","last_active":"2023-12-03T11:42:33.497138+0000","last_peered":"2023-12-03T11:42:33.497138+0000","last_clean":"2023-12-03T11:42:33.497138+0000","last_became_active":"2023-12-03T11:42:31.988458+0000","last_became_peered":"2023-12-03T11:42:31.988458+0000","last_unstale":"2023-12-03T11:42:33.497138+0000","last_undegraded":"2023-12-03T11:42:33.497138+0000","last_fullsized":"2023-12-03T11:42:33.497138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:51:24.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497186+0000","last_change":"2023-12-03T11:42:31.988588+0000","last_active":"2023-12-03T11:42:33.497186+0000","last_peered":"2023-12-03T11:42:33.497186+0000","last_clean":"2023-12-03T11:42:33.497186+0000","last_became_active":"2023-12-03T11:42:31.988232+0000","last_became_peered":"2023-12-03T11:42:31.988232+0000","last_unstale":"2023-12-03T11:42:33.497186+0000","last_undegraded":"2023-12-03T11:42:33.497186+0000","last_fullsized":"2023-12-03T11:42:33.497186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:02:23.877253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:42:31.984184+0000","last_change":"2023-12-03T11:42:25.991263+0000","last_active":"2023-12-03T11:42:31.984184+0000","last_peered":"2023-12-03T11:42:31.984184+0000","last_clean":"2023-12-03T11:42:31.984184+0000","last_became_active":"2023-12-03T11:42:25.990133+0000","last_became_peered":"2023-12-03T11:42:25.990133+0000","last_unstale":"2023-12-03T11:42:31.984184+0000","last_undegraded":"2023-12-03T11:42:31.984184+0000","last_fullsized":"2023-12-03T11:42:31.984184+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:24.953588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:24.953588+0000","last_clean_scrub_stamp":"2023-12-03T11:42:24.953588+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:30:30.854189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-03T11:42:35.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:42:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:36.028 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-12-03T11:42:36.040 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-03T11:42:34.122603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218812,"kb_used_data":2628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755908,"statfs":{"total":773094113280,"available":772870049792,"internally_reserved":0,"allocated":2691072,"data_stored":1667812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000924"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.496368+0000","last_change":"2023-12-03T11:42:31.987483+0000","last_active":"2023-12-03T11:42:33.496368+0000","last_peered":"2023-12-03T11:42:33.496368+0000","last_clean":"2023-12-03T11:42:33.496368+0000","last_became_active":"2023-12-03T11:42:31.987164+0000","last_became_peered":"2023-12-03T11:42:31.987164+0000","last_unstale":"2023-12-03T11:42:33.496368+0000","last_undegraded":"2023-12-03T11:42:33.496368+0000","last_fullsized":"2023-12-03T11:42:33.496368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T15:35:59.006579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.000061+0000","last_change":"2023-12-03T11:42:31.986643+0000","last_active":"2023-12-03T11:42:33.000061+0000","last_peered":"2023-12-03T11:42:33.000061+0000","last_clean":"2023-12-03T11:42:33.000061+0000","last_became_active":"2023-12-03T11:42:31.986407+0000","last_became_peered":"2023-12-03T11:42:31.986407+0000","last_unstale":"2023-12-03T11:42:33.000061+0000","last_undegraded":"2023-12-03T11:42:33.000061+0000","last_fullsized":"2023-12-03T11:42:33.000061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:57:19.678143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497153+0000","last_change":"2023-12-03T11:42:32.001616+0000","last_active":"2023-12-03T11:42:33.497153+0000","last_peered":"2023-12-03T11:42:33.497153+0000","last_clean":"2023-12-03T11:42:33.497153+0000","last_became_active":"2023-12-03T11:42:32.001354+0000","last_became_peered":"2023-12-03T11:42:32.001354+0000","last_unstale":"2023-12-03T11:42:33.497153+0000","last_undegraded":"2023-12-03T11:42:33.497153+0000","last_fullsized":"2023-12-03T11:42:33.497153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:15:21.172972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:32.999951+0000","last_change":"2023-12-03T11:42:32.001658+0000","last_active":"2023-12-03T11:42:32.999951+0000","last_peered":"2023-12-03T11:42:32.999951+0000","last_clean":"2023-12-03T11:42:32.999951+0000","last_became_active":"2023-12-03T11:42:32.001291+0000","last_became_peered":"2023-12-03T11:42:32.001291+0000","last_unstale":"2023-12-03T11:42:32.999951+0000","last_undegraded":"2023-12-03T11:42:32.999951+0000","last_fullsized":"2023-12-03T11:42:32.999951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T20:30:24.051910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497183+0000","last_change":"2023-12-03T11:42:31.985722+0000","last_active":"2023-12-03T11:42:33.497183+0000","last_peered":"2023-12-03T11:42:33.497183+0000","last_clean":"2023-12-03T11:42:33.497183+0000","last_became_active":"2023-12-03T11:42:31.985363+0000","last_became_peered":"2023-12-03T11:42:31.985363+0000","last_unstale":"2023-12-03T11:42:33.497183+0000","last_undegraded":"2023-12-03T11:42:33.497183+0000","last_fullsized":"2023-12-03T11:42:33.497183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:54:56.421628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497176+0000","last_change":"2023-12-03T11:42:32.109247+0000","last_active":"2023-12-03T11:42:33.497176+0000","last_peered":"2023-12-03T11:42:33.497176+0000","last_clean":"2023-12-03T11:42:33.497176+0000","last_became_active":"2023-12-03T11:42:32.108940+0000","last_became_peered":"2023-12-03T11:42:32.108940+0000","last_unstale":"2023-12-03T11:42:33.497176+0000","last_undegraded":"2023-12-03T11:42:33.497176+0000","last_fullsized":"2023-12-03T11:42:33.497176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T12:23:57.934082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497138+0000","last_change":"2023-12-03T11:42:31.988766+0000","last_active":"2023-12-03T11:42:33.497138+0000","last_peered":"2023-12-03T11:42:33.497138+0000","last_clean":"2023-12-03T11:42:33.497138+0000","last_became_active":"2023-12-03T11:42:31.988458+0000","last_became_peered":"2023-12-03T11:42:31.988458+0000","last_unstale":"2023-12-03T11:42:33.497138+0000","last_undegraded":"2023-12-03T11:42:33.497138+0000","last_fullsized":"2023-12-03T11:42:33.497138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:51:24.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:42:33.497186+0000","last_change":"2023-12-03T11:42:31.988588+0000","last_active":"2023-12-03T11:42:33.497186+0000","last_peered":"2023-12-03T11:42:33.497186+0000","last_clean":"2023-12-03T11:42:33.497186+0000","last_became_active":"2023-12-03T11:42:31.988232+0000","last_became_peered":"2023-12-03T11:42:31.988232+0000","last_unstale":"2023-12-03T11:42:33.497186+0000","last_undegraded":"2023-12-03T11:42:33.497186+0000","last_fullsized":"2023-12-03T11:42:33.497186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:30.973058+0000","last_clean_scrub_stamp":"2023-12-03T11:42:30.973058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:02:23.877253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:42:31.984184+0000","last_change":"2023-12-03T11:42:25.991263+0000","last_active":"2023-12-03T11:42:31.984184+0000","last_peered":"2023-12-03T11:42:31.984184+0000","last_clean":"2023-12-03T11:42:31.984184+0000","last_became_active":"2023-12-03T11:42:25.990133+0000","last_became_peered":"2023-12-03T11:42:25.990133+0000","last_unstale":"2023-12-03T11:42:31.984184+0000","last_undegraded":"2023-12-03T11:42:31.984184+0000","last_fullsized":"2023-12-03T11:42:31.984184+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:42:24.953588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:42:24.953588+0000","last_clean_scrub_stamp":"2023-12-03T11:42:24.953588+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:30:30.854189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-03T11:42:36.041 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-03T11:42:36.041 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-03T11:42:36.041 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-03T11:42:36.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-03T11:42:36.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:42:36.354 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-03T11:42:36.367 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-03T11:42:36.367 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-12-03T11:42:36.378 DEBUG:teuthology.task:Applying overrides for task openssl_keys: {'rgw.client.0': {'ca': 'root', 'client': 'client.0', 'embed-key': True}, 'root': {'client': 'client.0', 'cn': 'teuthology', 'install': ['client.0'], 'key-type': 'rsa:4096'}} 2023-12-03T11:42:36.379 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p /home/ubuntu/cephtest/ca 2023-12-03T11:42:36.394 DEBUG:teuthology.orchestra.run.smithi116:> openssl req -x509 -nodes -newkey rsa:4096 -keyout /home/ubuntu/cephtest/ca/root.key -days 365 -out /home/ubuntu/cephtest/ca/root.crt -subj /CN=teuthology 2023-12-03T11:42:37.261 INFO:teuthology.orchestra.run.smithi116.stderr:.+....+...+.....+......+.......+........+.+......+..+......+...............+....+..+.+...+..+............+...+....+......+..+......+.+.....+..................+...+.......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.......+.....+....+......+............+..+...+.......+.....+....+............+.........+......+........+.+..................+...+.........+...+......+..+.........+.+...+..+............+.+.....+.+...+...+..+.+.....+.........+............+.+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+...+......+.+...............+......+.....+..........+..............+....+...............+.........+........+...+.+.....+...+.......+........+..........+..+...+......+......+.+.....+......................+.....+..........+...........+....+.................+.+...+...........+.+.............................+.+...+..+....+.........+..............+...+.........+...+.....................+.........+.............+........+....+..+.............+..+.........+....+......+..............+.......+......+.........+...+........+...+.+.....+....+...........+.+...+.................+....+...+...+..+...+...+.+...+...+...+...........+.+.....+....+.....+.+..............+.+...............+..+.+..+.........................+..+...+.......+..+....+......+.....+....+..+.......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2023-12-03T11:42:37.588 INFO:teuthology.orchestra.run.smithi116.stderr:.......+...+...........+...+.+.........+......+.....+.........+............+.+.....+....+...+............+.....+....+..+.......+..+.+..+...+.......+.....+......+....+...+.........+..............+......+.+..+...+...+.........+......+.+.........+..+.........+.+............+...........+.+...+......+......+.....+.+...+......+.....+...+...+..........+..+...+...............+.........+....+.........+.....+.+........................+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...................................+.........+..........+..+..........+...+..+............+.+.....+......................+..............+...+.+.....+....+........+....+......+.....+.........+.+..+...+............+.............+.....+.............+......+....................+...+.......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2023-12-03T11:42:37.602 INFO:teuthology.orchestra.run.smithi116.stderr:----- 2023-12-03T11:42:37.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /home/ubuntu/cephtest/ca/root.crt /usr/share/pki/ca-trust-source/anchors/root.crt 2023-12-03T11:42:37.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-ca-trust 2023-12-03T11:42:38.499 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p /home/ubuntu/cephtest/ca 2023-12-03T11:42:38.514 DEBUG:teuthology.orchestra.run.smithi116:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/rgw.client.0.key -out /home/ubuntu/cephtest/ca/rgw.client.0.csr -subj /CN=smithi116.front.sepia.ceph.com 2023-12-03T11:42:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:........+...+.+.....+.+........+....+...+.....+.........+.........+....+...............+.....+......+.+.....+...+.+..+............+.......+.........+..+...+......+...+.+..+.......+.....+......+.+......+...+...............+.....+......+...............+...+...+...+....+...+........+......+.+...+......+.....+..........+...+...+..+......+.........+.+......+........+............+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+..+....+.....+...+.......+...+...+.....+...+.+...+...+..+.....................+.+...+.....+...+.............+..+.........+....+...........+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...........+.+...+...+........+.+......+...+..+.........................+............+..+.+..+...............+...+.......+...............+.....+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2023-12-03T11:42:38.782 INFO:teuthology.orchestra.run.smithi116.stderr:...+.....+...+.+.....+....+..+.+........+....+...+...+..+...+.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+...+.+...+......+...........+.........+.+...........+...+.........+......+....+...+........+......+..................+.+............+..+...+....+...+...........+.+........+......+.+......+....................+.......+.....+............+....+......+...+...+.....+.......+..+....+.....+.+.....+.+...+............+..+...+.........+.+...+..+.........+..........+...+.....+..........+.....+......+....+...+..+.............+...+..+...+.+...+..+.+............+..+.+........+......+...............+...................+..+....+...+..+.+..+......+...................+..+.+..+...+....+..............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2023-12-03T11:42:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:----- 2023-12-03T11:42:38.788 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:42:38.788 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ca/rgw.client.0.ext 2023-12-03T11:42:38.844 DEBUG:teuthology.orchestra.run.smithi116:> openssl x509 -req -in /home/ubuntu/cephtest/ca/rgw.client.0.csr -CA /home/ubuntu/cephtest/ca/root.crt -CAkey /home/ubuntu/cephtest/ca/root.key -CAcreateserial -out /home/ubuntu/cephtest/ca/rgw.client.0.crt -days 365 -sha256 -extfile /home/ubuntu/cephtest/ca/rgw.client.0.ext 2023-12-03T11:42:38.920 INFO:teuthology.orchestra.run.smithi116.stderr:Certificate request self-signature ok 2023-12-03T11:42:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:subject=CN = smithi116.front.sepia.ceph.com 2023-12-03T11:42:38.929 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/ca/rgw.client.0.csr /home/ubuntu/cephtest/ca/root.srl /home/ubuntu/cephtest/ca/rgw.client.0.ext 2023-12-03T11:42:38.984 DEBUG:teuthology.orchestra.run.smithi116:> openssl verify -CAfile /home/ubuntu/cephtest/ca/root.crt /home/ubuntu/cephtest/ca/rgw.client.0.crt 2023-12-03T11:42:39.042 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/ca/rgw.client.0.crt: OK 2023-12-03T11:42:39.042 DEBUG:teuthology.orchestra.run.smithi116:> cat /home/ubuntu/cephtest/ca/rgw.client.0.key >> /home/ubuntu/cephtest/ca/rgw.client.0.crt 2023-12-03T11:42:39.097 INFO:teuthology.run_tasks:Running task rgw... 2023-12-03T11:42:39.111 DEBUG:tasks.rgw:config is {'client.0': {'ssl certificate': 'rgw.client.0'}} 2023-12-03T11:42:39.111 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-12-03T11:42:39.111 INFO:tasks.rgw:Creating data pools 2023-12-03T11:42:39.111 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-12-03T11:42:39.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd erasure-code-profile set client.0 k=2 m=1 crush-failure-domain=osd --cluster ceph 2023-12-03T11:42:40.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2023-12-03T11:42:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.data' created 2023-12-03T11:42:42.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-12-03T11:42:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-12-03T11:42:43.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-12-03T11:42:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.index' created 2023-12-03T11:42:44.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-12-03T11:42:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-12-03T11:42:45.076 DEBUG:tasks.rgw:Pools created 2023-12-03T11:42:45.076 INFO:tasks.rgw:Configuring compression type = random 2023-12-03T11:42:45.076 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:42:45.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:42:45.076 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:42:45.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:45.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:45.178+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.056+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.056+0000 7f2c941ab800 20 realm 2023-12-03T11:42:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.056+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.057+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.057+0000 7f2c941ab800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:42:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.057+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.058+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.058+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.060+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.060+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.060+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.060+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.061+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.061+0000 7f2c941ab800 10 Creating default zonegroup 2023-12-03T11:42:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.061+0000 7f2c941ab800 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-12-03T11:42:47.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.072+0000 7f2c91cab640 0 -- 172.21.15.116:0/2559090081 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f2c7c064730 msgr2=0x7f2c7c0410b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:42:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.075+0000 7f2c941ab800 10 WARNING: driver->list_zones() returned r=-2 2023-12-03T11:42:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.075+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.075+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 10 could not read realm id: (2) No such file or directory 2023-12-03T11:42:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 10 WARNING: failed to set zone as default, r=-22 2023-12-03T11:42:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.080+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.087+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.091+0000 7f2c941ab800 20 rados_obj.operate() r=0 bl.length=874 2023-12-03T11:42:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.091+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.092+0000 7f2c941ab800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:42:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.092+0000 7f2c941ab800 10 Using default name default 2023-12-03T11:42:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.092+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.092+0000 7f2c941ab800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:42:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.092+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c924ac640 0 -- 172.21.15.116:0/2559090081 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x56533eff1010 msgr2=0x56533eff3400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 rados_obj.operate() r=0 bl.length=874 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 zone default found 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 4 Realm: () 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 zonegroup default 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:47.093+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:49.063+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:49.063+0000 7f2c941ab800 20 rados->read ofs=0 len=0 2023-12-03T11:42:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:49.068+0000 7f2c941ab800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:49.068+0000 7f2c941ab800 20 started sync module instance, tier type = 2023-12-03T11:42:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:49.068+0000 7f2c941ab800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:42:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=0 2023-12-03T11:42:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=1 2023-12-03T11:42:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=2 2023-12-03T11:42:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=3 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=4 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=5 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.102+0000 7f2c941ab800 20 add_watcher() i=6 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.103+0000 7f2c941ab800 20 add_watcher() i=7 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.103+0000 7f2c941ab800 2 all 8 watchers are set, enabling cache 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-12-03T11:42:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-12-03T11:42:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-12-03T11:42:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.104+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2023-12-03T11:42:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.105+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2023-12-03T11:42:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.106+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2023-12-03T11:42:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.107+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.108+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2023-12-03T11:42:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.109+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2023-12-03T11:42:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.110+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.111+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2023-12-03T11:42:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.112+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2023-12-03T11:42:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.113+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.114+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2023-12-03T11:42:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2023-12-03T11:42:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.115+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.116+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2023-12-03T11:42:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.117+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2023-12-03T11:42:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.118+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.119+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.120+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.121+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2023-12-03T11:42:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2023-12-03T11:42:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2023-12-03T11:42:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.122+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.123+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2023-12-03T11:42:51.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.124+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2023-12-03T11:42:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.125+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.126+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2023-12-03T11:42:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.127+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2023-12-03T11:42:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.128+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.129+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.130+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2023-12-03T11:42:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.131+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2023-12-03T11:42:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2023-12-03T11:42:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2023-12-03T11:42:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.132+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2023-12-03T11:42:51.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.133+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.134+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2023-12-03T11:42:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.135+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.136+0000 7f2c941ab800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.136+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.137+0000 7f2c941ab800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.140+0000 7f2c577e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.140+0000 7f2c941ab800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:51.140+0000 7f2c941ab800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.084+0000 7f2c941ab800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-12-03T11:42:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.084+0000 7f2c941ab800 5 note: GC not initialized 2023-12-03T11:42:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.084+0000 7f2c941ab800 20 init_complete bucket index max shards: 11 2023-12-03T11:42:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.085+0000 7f2c941ab800 20 Filter name: none 2023-12-03T11:42:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.085+0000 7f2bbe7fc640 20 reqs_thread_entry: start 2023-12-03T11:42:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.085+0000 7f2baffff640 20 reqs_thread_entry: start 2023-12-03T11:42:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.111+0000 7f2c577e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.113+0000 7f2c941ab800 20 remove_watcher() i=0 2023-12-03T11:42:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.113+0000 7f2c941ab800 2 removed watcher, disabling cache 2023-12-03T11:42:54.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.122+0000 7f2c941ab800 20 remove_watcher() i=1 2023-12-03T11:42:54.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.123+0000 7f2c941ab800 20 remove_watcher() i=2 2023-12-03T11:42:54.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.124+0000 7f2c941ab800 20 remove_watcher() i=3 2023-12-03T11:42:54.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.126+0000 7f2c941ab800 20 remove_watcher() i=4 2023-12-03T11:42:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.127+0000 7f2c941ab800 20 remove_watcher() i=5 2023-12-03T11:42:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.128+0000 7f2c941ab800 20 remove_watcher() i=6 2023-12-03T11:42:54.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.129+0000 7f2c941ab800 20 remove_watcher() i=7 2023-12-03T11:42:54.213 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2023-12-03T11:42:54.213 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:42:54.214 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-12-03T11:42:54.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-12-03T11:42:54.214 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-12-03T11:42:54.264 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.264 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.274+0000 7fce09522800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.274+0000 7fce09522800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.277+0000 7fcd42ffd640 20 reqs_thread_entry: start 2023-12-03T11:42:54.289 INFO:teuthology.orchestra.run.smithi116.stdout:{"id":"21f2df23-11b5-4c91-821f-a8e55156195e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:42:54.290 DEBUG:tasks.util.rgw: json result: {'id': '21f2df23-11b5-4c91-821f-a8e55156195e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:42:54.290 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:42:54.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:42:54.290 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:42:54.331 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.331 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.340+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 realm 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.342+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.343+0000 7f7d8df20800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:42:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.343+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.344+0000 7f7d8df20800 20 rados_obj.operate() r=0 bl.length=890 2023-12-03T11:42:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.344+0000 7f7d8df20800 20 searching for the correct realm 2023-12-03T11:42:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.354+0000 7f7d89c7e640 0 -- 172.21.15.116:0/1793161261 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557725baa170 msgr2=0x557725bac560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:42:54.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.356+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:42:54.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:42:54.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 zone default found 2023-12-03T11:42:54.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 4 Realm: () 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 zonegroup default 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.357+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.358+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.358+0000 7f7d8df20800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.358+0000 7f7d8df20800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.358+0000 7f7d8df20800 20 started sync module instance, tier type = 2023-12-03T11:42:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.358+0000 7f7d8df20800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:42:54.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.366+0000 7f7d89c7e640 0 -- 172.21.15.116:0/1793161261 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f7d84071040 msgr2=0x7f7d8405c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:42:54.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=0 2023-12-03T11:42:54.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=1 2023-12-03T11:42:54.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=2 2023-12-03T11:42:54.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=3 2023-12-03T11:42:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=4 2023-12-03T11:42:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=5 2023-12-03T11:42:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.370+0000 7f7d8df20800 20 add_watcher() i=6 2023-12-03T11:42:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.371+0000 7f7d8df20800 20 add_watcher() i=7 2023-12-03T11:42:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.371+0000 7f7d8df20800 2 all 8 watchers are set, enabling cache 2023-12-03T11:42:54.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.373+0000 7f7d4f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:54.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.373+0000 7f7d8df20800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.373+0000 7f7d8df20800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.377+0000 7f7d8df20800 5 note: GC not initialized 2023-12-03T11:42:54.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.378+0000 7f7cbe7fc640 20 reqs_thread_entry: start 2023-12-03T11:42:54.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.378+0000 7f7d8df20800 20 init_complete bucket index max shards: 11 2023-12-03T11:42:54.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.378+0000 7f7d8df20800 20 Filter name: none 2023-12-03T11:42:54.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.378+0000 7f7caffff640 20 reqs_thread_entry: start 2023-12-03T11:42:54.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.387+0000 7f7d4f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:54.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.389+0000 7f7d8df20800 20 remove_watcher() i=0 2023-12-03T11:42:54.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.389+0000 7f7d8df20800 2 removed watcher, disabling cache 2023-12-03T11:42:54.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.391+0000 7f7d8df20800 20 remove_watcher() i=1 2023-12-03T11:42:54.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.392+0000 7f7d8df20800 20 remove_watcher() i=2 2023-12-03T11:42:54.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.393+0000 7f7d8df20800 20 remove_watcher() i=3 2023-12-03T11:42:54.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.394+0000 7f7d8df20800 20 remove_watcher() i=4 2023-12-03T11:42:54.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.395+0000 7f7d8df20800 20 remove_watcher() i=5 2023-12-03T11:42:54.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.396+0000 7f7d8df20800 20 remove_watcher() i=6 2023-12-03T11:42:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.397+0000 7f7d8df20800 20 remove_watcher() i=7 2023-12-03T11:42:54.402 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2023-12-03T11:42:54.403 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:42:54.403 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2023-12-03T11:42:54.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2023-12-03T11:42:54.403 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --placement-inline-data false 2023-12-03T11:42:54.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.499+0000 7efcd3d20800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.499+0000 7efcd3d20800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.502+0000 7efc0dffb640 20 reqs_thread_entry: start 2023-12-03T11:42:54.514 INFO:teuthology.orchestra.run.smithi116.stdout:{"id":"21f2df23-11b5-4c91-821f-a8e55156195e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:42:54.514 DEBUG:tasks.util.rgw: json result: {'id': '21f2df23-11b5-4c91-821f-a8e55156195e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:42:54.514 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:42:54.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:42:54.514 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:42:54.555 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.555 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.564+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 realm 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.566+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.567+0000 7fad2a325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:42:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.567+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.568+0000 7fad2a325800 20 rados_obj.operate() r=0 bl.length=890 2023-12-03T11:42:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.568+0000 7fad2a325800 20 searching for the correct realm 2023-12-03T11:42:54.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad25f54640 0 -- 172.21.15.116:0/2046831484 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x56450d693e20 msgr2=0x56450d6962e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:42:54.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:42:54.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.579+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:42:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:42:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 zone default found 2023-12-03T11:42:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 4 Realm: () 2023-12-03T11:42:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:42:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:42:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:42:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 zonegroup default 2023-12-03T11:42:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.580+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.581+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.581+0000 7fad2a325800 20 rados->read ofs=0 len=0 2023-12-03T11:42:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.581+0000 7fad2a325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:42:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.581+0000 7fad2a325800 20 started sync module instance, tier type = 2023-12-03T11:42:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.581+0000 7fad2a325800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:42:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.591+0000 7fad1d753640 0 -- 172.21.15.116:0/2046831484 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x56450d696db0 msgr2=0x56450d69b020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=0 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=1 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=2 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=3 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=4 2023-12-03T11:42:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.594+0000 7fad2a325800 20 add_watcher() i=5 2023-12-03T11:42:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.599+0000 7fad2a325800 20 add_watcher() i=6 2023-12-03T11:42:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.599+0000 7fad2a325800 20 add_watcher() i=7 2023-12-03T11:42:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.599+0000 7fad2a325800 2 all 8 watchers are set, enabling cache 2023-12-03T11:42:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.601+0000 7faceb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.601+0000 7fad2a325800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.601+0000 7fad2a325800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.606+0000 7fad2a325800 5 note: GC not initialized 2023-12-03T11:42:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.606+0000 7fac5a7fc640 20 reqs_thread_entry: start 2023-12-03T11:42:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.606+0000 7fad2a325800 20 init_complete bucket index max shards: 11 2023-12-03T11:42:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.606+0000 7fad2a325800 20 Filter name: none 2023-12-03T11:42:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.606+0000 7fac4bfff640 20 reqs_thread_entry: start 2023-12-03T11:42:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.616+0000 7faceb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:42:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.618+0000 7fad2a325800 20 remove_watcher() i=0 2023-12-03T11:42:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.618+0000 7fad2a325800 2 removed watcher, disabling cache 2023-12-03T11:42:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.620+0000 7fad2a325800 20 remove_watcher() i=1 2023-12-03T11:42:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.621+0000 7fad2a325800 20 remove_watcher() i=2 2023-12-03T11:42:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.622+0000 7fad2a325800 20 remove_watcher() i=3 2023-12-03T11:42:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.623+0000 7fad2a325800 20 remove_watcher() i=4 2023-12-03T11:42:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.625+0000 7fad2a325800 20 remove_watcher() i=5 2023-12-03T11:42:54.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.626+0000 7fad2a325800 20 remove_watcher() i=6 2023-12-03T11:42:54.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.627+0000 7fad2a325800 20 remove_watcher() i=7 2023-12-03T11:42:54.633 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2023-12-03T11:42:54.633 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:42:54.633 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2023-12-03T11:42:54.633 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-12-03T11:42:54.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-12-03T11:42:54.634 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2023-12-03T11:42:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.729+0000 7f258668c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.729+0000 7f258668c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.732+0000 7f24c27fc640 20 reqs_thread_entry: start 2023-12-03T11:42:54.743 INFO:teuthology.orchestra.run.smithi116.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2023-12-03T11:42:54.743 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2023-12-03T11:42:54.743 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-12-03T11:42:54.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-12-03T11:42:54.743 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2023-12-03T11:42:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.801+0000 7f2f9612c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.801+0000 7f2f9612c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.805+0000 7f2ecf7fe640 20 reqs_thread_entry: start 2023-12-03T11:42:54.815 INFO:teuthology.orchestra.run.smithi116.stdout:{"id":"21f2df23-11b5-4c91-821f-a8e55156195e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:42:54.816 DEBUG:tasks.util.rgw: json result: {'id': '21f2df23-11b5-4c91-821f-a8e55156195e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:42:54.816 INFO:tasks.rgw:Configuring storage class type = FROZEN 2023-12-03T11:42:54.816 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-12-03T11:42:54.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-12-03T11:42:54.816 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2023-12-03T11:42:54.862 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.862 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.872+0000 7f460c920800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.872+0000 7f460c920800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.875+0000 7f45467fc640 20 reqs_thread_entry: start 2023-12-03T11:42:54.885 INFO:teuthology.orchestra.run.smithi116.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2023-12-03T11:42:54.885 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2023-12-03T11:42:54.885 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-12-03T11:42:54.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-12-03T11:42:54.885 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2023-12-03T11:42:54.935 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:42:54.935 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:42:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.944+0000 7f86f1511800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:42:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.944+0000 7f86f1511800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:42:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:42:54.948+0000 7f86357fa640 20 reqs_thread_entry: start 2023-12-03T11:42:54.959 INFO:teuthology.orchestra.run.smithi116.stdout:{"id":"21f2df23-11b5-4c91-821f-a8e55156195e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:42:54.959 DEBUG:tasks.util.rgw: json result: {'id': '21f2df23-11b5-4c91-821f-a8e55156195e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:42:54.959 INFO:tasks.rgw:Starting rgw... 2023-12-03T11:42:54.959 INFO:tasks.rgw:rgw client.0 config is {'ssl certificate': 'rgw.client.0'} 2023-12-03T11:42:54.959 INFO:tasks.rgw:Using beast as radosgw frontend 2023-12-03T11:42:54.960 INFO:tasks.rgw.client.0:Restarting daemon 2023-12-03T11:42:54.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast ssl_certificate=/home/ubuntu/cephtest/ca/rgw.client.0.crt ssl_port=443' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-12-03T11:42:54.962 INFO:tasks.rgw.client.0:Started 2023-12-03T11:42:54.962 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on https://smithi116.front.sepia.ceph.com:443/ 2023-12-03T11:42:54.962 DEBUG:teuthology.orchestra.run.smithi116:> curl https://smithi116.front.sepia.ceph.com:443/ 2023-12-03T11:42:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-03T11:42:54.989 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-03T11:42:54.989 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2023-12-03T11:42:54.989 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-03T11:42:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:curl: (7) Failed to connect to smithi116.front.sepia.ceph.com port 443: Connection refused 2023-12-03T11:42:55.411 INFO:tasks.rgw.client.0.smithi116.stdout:2023-12-03T11:42:55.409+0000 7f74e7374880 -1 ssl_private_key was not found: rgw/cert//default.key 2023-12-03T11:42:55.990 DEBUG:teuthology.orchestra.run.smithi116:> curl https://smithi116.front.sepia.ceph.com:443/ 2023-12-03T11:42:56.009 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-03T11:42:56.009 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2023-12-03T11:42:56.078 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 3101 0 --:--:-- --:--:-- --:--:-- 3147 2023-12-03T11:42:56.079 INFO:teuthology.orchestra.run.smithi116.stdout:anonymous 2023-12-03T11:42:56.079 INFO:teuthology.run_tasks:Running task tox... 2023-12-03T11:42:56.090 INFO:tasks.tox:Deploying tox from pip... 2023-12-03T11:42:56.091 DEBUG:teuthology.orchestra.run.smithi116:> python3 -m venv /home/ubuntu/cephtest/tox-venv 2023-12-03T11:42:58.543 DEBUG:teuthology.orchestra.run.smithi116:> source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==3.15.0 2023-12-03T11:42:59.555 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting tox==3.15.0 2023-12-03T11:42:59.664 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading tox-3.15.0-py2.py3-none-any.whl (82 kB) 2023-12-03T11:42:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting py<2,>=1.4.17 2023-12-03T11:42:59.956 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading py-1.11.0-py2.py3-none-any.whl (98 kB) 2023-12-03T11:43:00.177 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting six<2,>=1.14.0 2023-12-03T11:43:00.194 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:43:00.396 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting toml>=0.9.4 2023-12-03T11:43:00.413 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2023-12-03T11:43:00.624 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting pluggy<1,>=0.12.0 2023-12-03T11:43:00.641 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB) 2023-12-03T11:43:00.883 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting filelock<4,>=3.0.0 2023-12-03T11:43:00.900 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading filelock-3.13.1-py3-none-any.whl (11 kB) 2023-12-03T11:43:01.293 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 2023-12-03T11:43:01.312 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB) 2023-12-03T11:43:01.640 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting packaging>=14 2023-12-03T11:43:01.657 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading packaging-23.2-py3-none-any.whl (53 kB) 2023-12-03T11:43:01.902 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting distlib<1,>=0.3.7 2023-12-03T11:43:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB) 2023-12-03T11:43:02.153 INFO:teuthology.orchestra.run.smithi116.stdout:Collecting platformdirs<5,>=3.9.1 2023-12-03T11:43:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: Downloading platformdirs-4.0.0-py3-none-any.whl (17 kB) 2023-12-03T11:43:02.284 INFO:teuthology.orchestra.run.smithi116.stdout:Installing collected packages: platformdirs, filelock, distlib, virtualenv, toml, six, py, pluggy, packaging, tox 2023-12-03T11:43:03.101 INFO:teuthology.orchestra.run.smithi116.stdout:Successfully installed distlib-0.3.7 filelock-3.13.1 packaging-23.2 platformdirs-4.0.0 pluggy-0.13.1 py-1.11.0 six-1.16.0 toml-0.10.2 tox-3.15.0 virtualenv-20.25.0 2023-12-03T11:43:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:WARNING: You are using pip version 21.2.3; however, version 23.3.1 is available. 2023-12-03T11:43:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:You should consider upgrading via the '/home/ubuntu/cephtest/tox-venv/bin/python3 -m pip install --upgrade pip' command. 2023-12-03T11:43:03.809 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:43:03.822 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:43:03.823 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:43:03.823 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:43:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:43:03.839 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:43:03.839 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:43:03.893 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:43:03.894 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:43:03.949 INFO:tasks.workunit:timeout=3h 2023-12-03T11:43:03.949 INFO:tasks.workunit:cleanup=True 2023-12-03T11:43:03.949 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:43:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:44:08.279 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:44:08.279 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.280 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2023-12-03T11:44:08.281 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.281 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:44:08.281 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:08.281 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:44:08.282 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:44:08.349 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:44:08.351 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:44:08.351 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:44:09.012 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:44:09.076 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:44:09.117 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:44:09.119 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:44:09.119 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:44:09.159 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:44:09.162 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:44:09.163 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:44:09.218 INFO:tasks.workunit:Running workunits matching rgw/run-bucket-check.sh on client.0... 2023-12-03T11:44:09.221 INFO:tasks.workunit:Running workunit rgw/run-bucket-check.sh... 2023-12-03T11:44:09.222 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/run-bucket-check.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-12-03T11:44:09.277 INFO:tasks.workunit.client.0.smithi116.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-12-03T11:44:09.278 INFO:tasks.workunit.client.0.smithi116.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:44:09.278 INFO:tasks.workunit.client.0.smithi116.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:44:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2023-12-03T11:44:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:++ deactivate nondestructive 2023-12-03T11:44:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:44:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ hash -r 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ unset VIRTUAL_ENV 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' '!' nondestructive = nondestructive ']' 2023-12-03T11:44:11.634 INFO:tasks.workunit.client.0.smithi116.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ export VIRTUAL_ENV 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ _OLD_VIRTUAL_PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ export PATH 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:44:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -z '' ']' 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:++ _OLD_VIRTUAL_PS1= 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:++ PS1='(rgw) ' 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:++ export PS1 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:++ hash -r 2023-12-03T11:44:11.636 INFO:tasks.workunit.client.0.smithi116.stderr:+ pip install pip --upgrade 2023-12-03T11:44:11.862 INFO:tasks.workunit.client.0.smithi116.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:44:12.313 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting pip 2023-12-03T11:44:12.835 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:44:13.746 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: pip 2023-12-03T11:44:13.750 INFO:tasks.workunit.client.0.smithi116.stdout: Attempting uninstall: pip 2023-12-03T11:44:13.750 INFO:tasks.workunit.client.0.smithi116.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:44:13.898 INFO:tasks.workunit.client.0.smithi116.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:44:13.903 INFO:tasks.workunit.client.0.smithi116.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:44:14.845 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed pip-23.3.1 2023-12-03T11:44:15.438 INFO:tasks.workunit.client.0.smithi116.stderr:+ pip install boto3 2023-12-03T11:44:16.343 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting boto3 2023-12-03T11:44:16.837 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:44:17.524 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:44:17.661 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:44:17.797 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:44:17.953 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:44:18.127 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:44:18.268 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:44:18.428 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:44:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:44:19.034 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 542.9 kB/s eta 0:00:00 2023-12-03T11:44:19.150 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:44:19.302 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:44:19.401 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 404.2 kB/s eta 0:00:00 2023-12-03T11:44:19.670 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:44:19.821 INFO:tasks.workunit.client.0.smithi116.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:44:20.045 INFO:tasks.workunit.client.0.smithi116.stdout:Downloading boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:44:20.127 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 1.6 MB/s eta 0:00:00 2023-12-03T11:44:20.255 INFO:tasks.workunit.client.0.smithi116.stdout:Downloading botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:44:21.154 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 17.3 MB/s eta 0:00:00 2023-12-03T11:44:21.290 INFO:tasks.workunit.client.0.smithi116.stdout:Downloading s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:44:21.353 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 kB 1.2 MB/s eta 0:00:00 2023-12-03T11:44:21.500 INFO:tasks.workunit.client.0.smithi116.stdout:Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:44:21.554 INFO:tasks.workunit.client.0.smithi116.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.8/143.8 kB 2.6 MB/s eta 0:00:00 2023-12-03T11:44:21.654 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:44:22.705 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:44:22.832 INFO:tasks.workunit.client.0.smithi116.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py 2023-12-03T11:44:22.966 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin user create --uid check-tester --display-name "Check Testing" --access-key OJODXSLNX4LUNHQG99PA --secret 3l6ffld34qaymfomuh832j94738aie2x4p2o8h6n 2023-12-03T11:44:22.999 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:22.999 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:23.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.011+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 realm 2023-12-03T11:44:23.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.013+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.014+0000 7fa2e9f26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.014+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.014+0000 7fa2e9f26800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:23.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.014+0000 7fa2e9f26800 20 searching for the correct realm 2023-12-03T11:44:23.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.025+0000 7fa2e5c61640 0 -- 172.21.15.116:0/1396912442 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55b09f372f60 msgr2=0x55b09f375350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.026+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 zone default found 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 4 Realm: () 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 zonegroup default 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.027+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.028+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.028+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.028+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.028+0000 7fa2e9f26800 20 started sync module instance, tier type = 2023-12-03T11:44:23.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.028+0000 7fa2e9f26800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:23.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.042+0000 7fa2e5460640 0 -- 172.21.15.116:0/1396912442 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55b09f377f40 msgr2=0x55b09f37a330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=0 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=1 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=2 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=3 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=4 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=5 2023-12-03T11:44:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.045+0000 7fa2e9f26800 20 add_watcher() i=6 2023-12-03T11:44:23.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.047+0000 7fa2e9f26800 20 add_watcher() i=7 2023-12-03T11:44:23.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.047+0000 7fa2e9f26800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.049+0000 7fa2ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.049+0000 7fa2e9f26800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.049+0000 7fa2e9f26800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa2e9f26800 5 note: GC not initialized 2023-12-03T11:44:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa21a7fc640 20 reqs_thread_entry: start 2023-12-03T11:44:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa2e9f26800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa2e9f26800 20 Filter name: none 2023-12-03T11:44:23.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa20bfff640 20 reqs_thread_entry: start 2023-12-03T11:44:23.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa2e9f26800 10 cache get: name=default.rgw.meta+users.uid+check-tester : miss 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.055+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 adding default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : miss 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x0 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 adding default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:44:23.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-12-03T11:44:23.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.056+0000 7fa2e9f26800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2e9f26800 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2e9f26800 10 moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2e9f26800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:check-tester, ofs0, ns] 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2c7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757248 cookie 94217073807328 notifier 4586 bl.length()=559 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2c7fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-12-03T11:44:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.058+0000 7fa2c7fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:44:23.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2e9f26800 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-12-03T11:44:23.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2e9f26800 10 moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:44:23.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2e9f26800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA, ofs0, ns] 2023-12-03T11:44:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2c7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757248 cookie 140337519656960 notifier 4586 bl.length()=186 2023-12-03T11:44:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2c7fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-12-03T11:44:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.060+0000 7fa2c7fff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:44:23.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.061+0000 7fa2ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.064+0000 7fa2e9f26800 20 remove_watcher() i=0 2023-12-03T11:44:23.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.064+0000 7fa2e9f26800 2 removed watcher, disabling cache 2023-12-03T11:44:23.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.066+0000 7fa2e9f26800 20 remove_watcher() i=1 2023-12-03T11:44:23.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.067+0000 7fa2e9f26800 20 remove_watcher() i=2 2023-12-03T11:44:23.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.068+0000 7fa2e9f26800 20 remove_watcher() i=3 2023-12-03T11:44:23.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.069+0000 7fa2e9f26800 20 remove_watcher() i=4 2023-12-03T11:44:23.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.070+0000 7fa2e9f26800 20 remove_watcher() i=5 2023-12-03T11:44:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.071+0000 7fa2e9f26800 20 remove_watcher() i=6 2023-12-03T11:44:23.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.072+0000 7fa2e9f26800 20 remove_watcher() i=7 2023-12-03T11:44:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.212 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.272 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.317 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.331 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.342 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.345 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.354 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.357 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.361 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.378 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.382 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.409 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.412 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.439 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.442 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:23.460 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:23.462 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: recalculated bucket check stats are correct 2023-12-03T11:44:23.462 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2023-12-03T11:44:23.491 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:23.491 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:23.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.503+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 realm 2023-12-03T11:44:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.505+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.506+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.506+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.506+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:23.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.506+0000 7ff73f931800 20 searching for the correct realm 2023-12-03T11:44:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73be36640 0 -- 172.21.15.116:0/3649891390 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7ff724054c00 msgr2=0x7ff724070900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73f931800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73f931800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73f931800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73f931800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.516+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 zone default found 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 4 Realm: () 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 zonegroup default 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.517+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 started sync module instance, tier type = 2023-12-03T11:44:23.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.518+0000 7ff73f931800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73b635640 0 -- 172.21.15.116:0/3649891390 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7ff72407d8f0 msgr2=0x7ff7240613a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=0 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=1 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=2 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=3 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=4 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=5 2023-12-03T11:44:23.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.535+0000 7ff73f931800 20 add_watcher() i=6 2023-12-03T11:44:23.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.537+0000 7ff73f931800 20 add_watcher() i=7 2023-12-03T11:44:23.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.537+0000 7ff73f931800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:23.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.540+0000 7ff7017e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.540+0000 7ff73f931800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:23.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.540+0000 7ff73f931800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff73f931800 5 note: GC not initialized 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff66ffff640 20 reqs_thread_entry: start 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff73f931800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff73f931800 20 Filter name: none 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff66dffb640 20 reqs_thread_entry: start 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff73f931800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:23.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.546+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.547+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.547+0000 7ff73f931800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.547+0000 7ff73f931800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.547+0000 7ff73f931800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.547+0000 7ff73f931800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:23.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:44:23.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.548+0000 7ff73f931800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:44:23.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.557+0000 7ff7017e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.560+0000 7ff73f931800 20 remove_watcher() i=0 2023-12-03T11:44:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.560+0000 7ff73f931800 2 removed watcher, disabling cache 2023-12-03T11:44:23.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.562+0000 7ff73f931800 20 remove_watcher() i=1 2023-12-03T11:44:23.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.563+0000 7ff73f931800 20 remove_watcher() i=2 2023-12-03T11:44:23.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.564+0000 7ff73f931800 20 remove_watcher() i=3 2023-12-03T11:44:23.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.565+0000 7ff73f931800 20 remove_watcher() i=4 2023-12-03T11:44:23.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.566+0000 7ff73f931800 20 remove_watcher() i=5 2023-12-03T11:44:23.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.567+0000 7ff73f931800 20 remove_watcher() i=6 2023-12-03T11:44:23.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.568+0000 7ff73f931800 20 remove_watcher() i=7 2023-12-03T11:44:23.576 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket check-bucket 2023-12-03T11:44:23.602 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:23.603 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:23.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.615+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 20 realm 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.616+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.617+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.617+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.617+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.617+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.618+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:23.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.618+0000 7fdedf126800 20 searching for the correct realm 2023-12-03T11:44:23.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedad23640 0 -- 172.21.15.116:0/3557042184 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55d23f172b70 msgr2=0x55d23f174f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:23.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:23.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.628+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 zone default found 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 4 Realm: () 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 zonegroup default 2023-12-03T11:44:23.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 started sync module instance, tier type = 2023-12-03T11:44:23.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.629+0000 7fdedf126800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:23.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.644+0000 7fdeda522640 0 -- 172.21.15.116:0/3557042184 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55d23f177b60 msgr2=0x55d23f179f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=0 2023-12-03T11:44:23.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=1 2023-12-03T11:44:23.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=2 2023-12-03T11:44:23.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=3 2023-12-03T11:44:23.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=4 2023-12-03T11:44:23.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.645+0000 7fdedf126800 20 add_watcher() i=5 2023-12-03T11:44:23.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.648+0000 7fdedf126800 20 add_watcher() i=6 2023-12-03T11:44:23.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.648+0000 7fdedf126800 20 add_watcher() i=7 2023-12-03T11:44:23.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.648+0000 7fdedf126800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:23.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.650+0000 7fde1bfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.650+0000 7fdedf126800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:23.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.650+0000 7fdedf126800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:23.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.655+0000 7fdedf126800 5 note: GC not initialized 2023-12-03T11:44:23.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fde12ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:23.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fdedf126800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fdedf126800 20 Filter name: none 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fde10ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.656+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:23.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:23.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.657+0000 7fdedf126800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:23.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.658+0000 7fdedf126800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.658+0000 7fdedf126800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.659+0000 7fde1bfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.661+0000 7fdedf126800 20 remove_watcher() i=0 2023-12-03T11:44:23.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.661+0000 7fdedf126800 2 removed watcher, disabling cache 2023-12-03T11:44:23.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.665+0000 7fdedf126800 20 remove_watcher() i=1 2023-12-03T11:44:23.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.667+0000 7fdedf126800 20 remove_watcher() i=2 2023-12-03T11:44:23.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.668+0000 7fdedf126800 20 remove_watcher() i=3 2023-12-03T11:44:23.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.669+0000 7fdedf126800 20 remove_watcher() i=4 2023-12-03T11:44:23.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.670+0000 7fdedf126800 20 remove_watcher() i=5 2023-12-03T11:44:23.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.671+0000 7fdedf126800 20 remove_watcher() i=6 2023-12-03T11:44:23.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.672+0000 7fdedf126800 20 remove_watcher() i=7 2023-12-03T11:44:23.678 INFO:tasks.workunit.client.0.smithi116.stderr:{'rgw.main': {'size': 54, 'size_actual': 24576, 'size_utilized': 109, 'size_kb': 1, 'size_kb_actual': 24, 'size_kb_utilized': 1, 'num_objects': 6}} 2023-12-03T11:44:23.679 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check unlinked does not report normal entries 2023-12-03T11:44:23.679 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:23.679 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:44:23.711 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:23.711 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:23.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.724+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 20 realm 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.725+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.726+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.726+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.726+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.726+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.727+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:23.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.727+0000 7fbee440f800 20 searching for the correct realm 2023-12-03T11:44:23.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.735+0000 7fbed970e640 0 -- 172.21.15.116:0/3429137277 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x557576c96940 msgr2=0x557576c98d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:23.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:23.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:23.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:23.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.737+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 zone default found 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 4 Realm: () 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:23.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 zonegroup default 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.738+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.739+0000 7fbee440f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.739+0000 7fbee440f800 20 started sync module instance, tier type = 2023-12-03T11:44:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.739+0000 7fbee440f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.746+0000 7fbee1f0f640 0 -- 172.21.15.116:0/3429137277 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fbedc05c0b0 msgr2=0x7fbedc0573d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:23.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=0 2023-12-03T11:44:23.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=1 2023-12-03T11:44:23.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=2 2023-12-03T11:44:23.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=3 2023-12-03T11:44:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=4 2023-12-03T11:44:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=5 2023-12-03T11:44:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.754+0000 7fbee440f800 20 add_watcher() i=6 2023-12-03T11:44:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.755+0000 7fbee440f800 20 add_watcher() i=7 2023-12-03T11:44:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.755+0000 7fbee440f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.756+0000 7fbea77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.757+0000 7fbee440f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.757+0000 7fbee440f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.762+0000 7fbee440f800 5 note: GC not initialized 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.762+0000 7fbe167fc640 20 reqs_thread_entry: start 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.762+0000 7fbee440f800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 20 Filter name: none 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbe07fff640 20 reqs_thread_entry: start 2023-12-03T11:44:23.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:23.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.763+0000 7fbee440f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:23.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.764+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 1 NOTICE: finished shard 1 (0 entries found) 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 1 NOTICE: finished shard 2 (0 entries found) 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:23.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 1 NOTICE: finished shard 3 (0 entries found) 2023-12-03T11:44:23.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.765+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:23.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.766+0000 7fbee440f800 1 NOTICE: finished shard 4 (0 entries found) 2023-12-03T11:44:23.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.766+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:23.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.766+0000 7fbee440f800 1 NOTICE: finished shard 5 (0 entries found) 2023-12-03T11:44:23.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.766+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.767+0000 7fbee440f800 1 NOTICE: finished shard 6 (0 entries found) 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.767+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.767+0000 7fbee440f800 1 NOTICE: finished shard 7 (0 entries found) 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.767+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 1 NOTICE: finished shard 8 (0 entries found) 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 1 NOTICE: finished shard 9 (0 entries found) 2023-12-03T11:44:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:23.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 1 NOTICE: finished shard 10 (0 entries found) 2023-12-03T11:44:23.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.768+0000 7fbee440f800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:44:23.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.769+0000 7fbea77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:23.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.771+0000 7fbee440f800 20 remove_watcher() i=0 2023-12-03T11:44:23.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.771+0000 7fbee440f800 2 removed watcher, disabling cache 2023-12-03T11:44:23.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.773+0000 7fbee440f800 20 remove_watcher() i=1 2023-12-03T11:44:23.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.774+0000 7fbee440f800 20 remove_watcher() i=2 2023-12-03T11:44:23.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.776+0000 7fbee440f800 20 remove_watcher() i=3 2023-12-03T11:44:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.777+0000 7fbee440f800 20 remove_watcher() i=4 2023-12-03T11:44:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.778+0000 7fbee440f800 20 remove_watcher() i=5 2023-12-03T11:44:23.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.779+0000 7fbee440f800 20 remove_watcher() i=6 2023-12-03T11:44:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:23.780+0000 7fbee440f800 20 remove_watcher() i=7 2023-12-03T11:44:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:44:24.164 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:44:25.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:25.539 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:25.593 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:25.595 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:44:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:25.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.641+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 20 realm 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.643+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.644+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.644+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.644+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:25.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.644+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.645+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:25.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.645+0000 7f072269d800 20 searching for the correct realm 2023-12-03T11:44:25.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.654+0000 7f07209a5640 0 -- 172.21.15.116:0/634058234 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f071401cc00 msgr2=0x7f071401b960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.657+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.658+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.658+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.658+0000 7f07209a5640 0 -- 172.21.15.116:0/634058234 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f071401cc00 msgr2=0x7f06fc07c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 zone default found 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 4 Realm: () 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 zonegroup default 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.860+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.861+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.861+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.861+0000 7f072269d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:25.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.861+0000 7f072269d800 20 started sync module instance, tier type = 2023-12-03T11:44:25.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.861+0000 7f072269d800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=0 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=1 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=2 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=3 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=4 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=5 2023-12-03T11:44:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.881+0000 7f072269d800 20 add_watcher() i=6 2023-12-03T11:44:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.882+0000 7f072269d800 20 add_watcher() i=7 2023-12-03T11:44:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.882+0000 7f072269d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:25.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.885+0000 7f06e67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:25.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.885+0000 7f072269d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:25.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.885+0000 7f072269d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:25.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f072269d800 5 note: GC not initialized 2023-12-03T11:44:25.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f06557fa640 20 reqs_thread_entry: start 2023-12-03T11:44:25.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f072269d800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:25.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f072269d800 20 Filter name: none 2023-12-03T11:44:25.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f064effd640 20 reqs_thread_entry: start 2023-12-03T11:44:25.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.892+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 rados->read ofs=0 len=0 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 INFO: main: starting shard=0 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.893+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 INFO: main: starting shard=1 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 INFO: main: starting shard=2 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.894+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 INFO: main: starting shard=3 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 INFO: main: starting shard=4 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.895+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.896+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.896+0000 7f072269d800 20 INFO: main: starting shard=5 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.896+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: starting shard=6 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: starting shard=7 2023-12-03T11:44:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:25.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 INFO: main: starting shard=8 2023-12-03T11:44:25.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.897+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.898+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.898+0000 7f072269d800 20 INFO: main: starting shard=9 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.898+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.899+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.899+0000 7f072269d800 20 INFO: main: starting shard=10 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.899+0000 7f072269d800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.899+0000 7f072269d800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.899+0000 7f072269d800 20 INFO: main: done 2023-12-03T11:44:25.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.900+0000 7f06e67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:25.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.904+0000 7f072269d800 20 remove_watcher() i=0 2023-12-03T11:44:25.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.904+0000 7f072269d800 2 removed watcher, disabling cache 2023-12-03T11:44:25.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.905+0000 7f072269d800 20 remove_watcher() i=1 2023-12-03T11:44:25.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.908+0000 7f072269d800 20 remove_watcher() i=2 2023-12-03T11:44:25.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.910+0000 7f072269d800 20 remove_watcher() i=3 2023-12-03T11:44:25.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.911+0000 7f072269d800 20 remove_watcher() i=4 2023-12-03T11:44:25.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.912+0000 7f072269d800 20 remove_watcher() i=5 2023-12-03T11:44:25.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.915+0000 7f072269d800 20 remove_watcher() i=6 2023-12-03T11:44:25.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:25.916+0000 7f072269d800 20 remove_watcher() i=7 2023-12-03T11:44:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:25.925 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:25.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:25.929 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:25.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:25.931 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:25.984 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:25.985 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:44:26.017 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:26.017 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.030+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 20 realm 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.031+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.032+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.032+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.033+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.033+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.033+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:26.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.033+0000 7f1413722800 20 searching for the correct realm 2023-12-03T11:44:26.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.042+0000 7f140f31c640 0 -- 172.21.15.116:0/3645745652 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55ed7c85ed00 msgr2=0x55ed7c861110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:26.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.044+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:26.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 zone default found 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 4 Realm: () 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 zonegroup default 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.045+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.046+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.046+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.046+0000 7f1413722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.046+0000 7f1413722800 20 started sync module instance, tier type = 2023-12-03T11:44:26.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.046+0000 7f1413722800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:26.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.061+0000 7f140fb1d640 0 -- 172.21.15.116:0/3645745652 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f140400c400 msgr2=0x7f140400c880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:26.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=0 2023-12-03T11:44:26.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=1 2023-12-03T11:44:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=2 2023-12-03T11:44:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=3 2023-12-03T11:44:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=4 2023-12-03T11:44:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.069+0000 7f1413722800 20 add_watcher() i=5 2023-12-03T11:44:26.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.071+0000 7f1413722800 20 add_watcher() i=6 2023-12-03T11:44:26.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.071+0000 7f1413722800 20 add_watcher() i=7 2023-12-03T11:44:26.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.071+0000 7f1413722800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:26.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.074+0000 7f13d57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.074+0000 7f1413722800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:26.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.074+0000 7f1413722800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1413722800 5 note: GC not initialized 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1343fff640 20 reqs_thread_entry: start 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1413722800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1413722800 20 Filter name: none 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1341ffb640 20 reqs_thread_entry: start 2023-12-03T11:44:26.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.080+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 INFO: main: starting shard=0 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.081+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:26.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: starting shard=1 2023-12-03T11:44:26.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:26.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: starting shard=2 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 INFO: main: starting shard=3 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.082+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 INFO: main: starting shard=4 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 INFO: main: starting shard=5 2023-12-03T11:44:26.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.083+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:26.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.084+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:44:26.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.084+0000 7f1413722800 20 INFO: main: starting shard=6 2023-12-03T11:44:26.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.084+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: starting shard=7 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: starting shard=8 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 INFO: main: starting shard=9 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.085+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.086+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.086+0000 7f1413722800 20 INFO: main: starting shard=10 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.086+0000 7f1413722800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.086+0000 7f1413722800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.086+0000 7f1413722800 20 INFO: main: done 2023-12-03T11:44:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.087+0000 7f13d57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.091+0000 7f1413722800 20 remove_watcher() i=0 2023-12-03T11:44:26.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.091+0000 7f1413722800 2 removed watcher, disabling cache 2023-12-03T11:44:26.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.092+0000 7f1413722800 20 remove_watcher() i=1 2023-12-03T11:44:26.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.095+0000 7f1413722800 20 remove_watcher() i=2 2023-12-03T11:44:26.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.097+0000 7f1413722800 20 remove_watcher() i=3 2023-12-03T11:44:26.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.098+0000 7f1413722800 20 remove_watcher() i=4 2023-12-03T11:44:26.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.099+0000 7f1413722800 20 remove_watcher() i=5 2023-12-03T11:44:26.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.101+0000 7f1413722800 20 remove_watcher() i=6 2023-12-03T11:44:26.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.102+0000 7f1413722800 20 remove_watcher() i=7 2023-12-03T11:44:26.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.110 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.113 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.116 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.173 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:26.174 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-12-03T11:44:26.202 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:26.202 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:26.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.216+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 realm 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.218+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.219+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.219+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.219+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:26.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.219+0000 7f944631f800 20 searching for the correct realm 2023-12-03T11:44:26.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.229+0000 7f9441794640 0 -- 172.21.15.116:0/2495096828 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55b7b55dbab0 msgr2=0x55b7b55ddea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:26.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.230+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 zone default found 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 4 Realm: () 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:26.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 zonegroup default 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.231+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.232+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.232+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.232+0000 7f944631f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.232+0000 7f944631f800 20 started sync module instance, tier type = 2023-12-03T11:44:26.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.232+0000 7f944631f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:26.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.240+0000 7f9442796640 0 -- 172.21.15.116:0/2495096828 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f943805f3e0 msgr2=0x7f94380588a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=0 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=1 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=2 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=3 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=4 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=5 2023-12-03T11:44:26.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.247+0000 7f944631f800 20 add_watcher() i=6 2023-12-03T11:44:26.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.248+0000 7f944631f800 20 add_watcher() i=7 2023-12-03T11:44:26.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.248+0000 7f944631f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:26.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.250+0000 7f94077e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.250+0000 7f944631f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:26.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.250+0000 7f944631f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:26.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f944631f800 5 note: GC not initialized 2023-12-03T11:44:26.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f93767fc640 20 reqs_thread_entry: start 2023-12-03T11:44:26.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f944631f800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:26.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f944631f800 20 Filter name: none 2023-12-03T11:44:26.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f9367fff640 20 reqs_thread_entry: start 2023-12-03T11:44:26.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.256+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 INFO: main: starting shard=0 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 INFO: main: starting shard=1 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.257+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 INFO: main: starting shard=2 2023-12-03T11:44:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 INFO: main: starting shard=3 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.258+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 INFO: main: starting shard=4 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 INFO: main: starting shard=5 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.259+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 INFO: main: starting shard=6 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 INFO: main: starting shard=7 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.260+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 INFO: main: starting shard=8 2023-12-03T11:44:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 INFO: main: starting shard=9 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.261+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.262+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.262+0000 7f944631f800 20 INFO: main: starting shard=10 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.262+0000 7f944631f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.262+0000 7f944631f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.262+0000 7f944631f800 20 INFO: main: done 2023-12-03T11:44:26.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.263+0000 7f94077e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.265+0000 7f944631f800 20 remove_watcher() i=0 2023-12-03T11:44:26.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.265+0000 7f944631f800 2 removed watcher, disabling cache 2023-12-03T11:44:26.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.266+0000 7f944631f800 20 remove_watcher() i=1 2023-12-03T11:44:26.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.267+0000 7f944631f800 20 remove_watcher() i=2 2023-12-03T11:44:26.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.268+0000 7f944631f800 20 remove_watcher() i=3 2023-12-03T11:44:26.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.269+0000 7f944631f800 20 remove_watcher() i=4 2023-12-03T11:44:26.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.270+0000 7f944631f800 20 remove_watcher() i=5 2023-12-03T11:44:26.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.273+0000 7f944631f800 20 remove_watcher() i=6 2023-12-03T11:44:26.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.274+0000 7f944631f800 20 remove_watcher() i=7 2023-12-03T11:44:26.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.282 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.286 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.341 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:26.343 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-12-03T11:44:26.371 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:26.371 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:26.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.384+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 realm 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.586+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.587+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.587+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.588+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.588+0000 7fa7028c0800 20 searching for the correct realm 2023-12-03T11:44:26.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.596+0000 7fa700bc1640 0 -- 172.21.15.116:0/3122116973 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x558b06bd43b0 msgr2=0x558b06bd67a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.598+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:26.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:26.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 zone default found 2023-12-03T11:44:26.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 4 Realm: () 2023-12-03T11:44:26.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 zonegroup default 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.599+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.600+0000 7fa7028c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.600+0000 7fa7028c0800 20 started sync module instance, tier type = 2023-12-03T11:44:26.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.600+0000 7fa7028c0800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:26.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.605+0000 7fa700bc1640 0 -- 172.21.15.116:0/3122116973 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7fa6e8071040 msgr2=0x7fa6e8071930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=0 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=1 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=2 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=3 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=4 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=5 2023-12-03T11:44:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.612+0000 7fa7028c0800 20 add_watcher() i=6 2023-12-03T11:44:26.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.613+0000 7fa7028c0800 20 add_watcher() i=7 2023-12-03T11:44:26.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.613+0000 7fa7028c0800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:26.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.615+0000 7fa6c67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.615+0000 7fa7028c0800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:26.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.615+0000 7fa7028c0800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.621+0000 7fa7028c0800 5 note: GC not initialized 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.621+0000 7fa6357fa640 20 reqs_thread_entry: start 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.621+0000 7fa7028c0800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.621+0000 7fa7028c0800 20 Filter name: none 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.621+0000 7fa62effd640 20 reqs_thread_entry: start 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 rados->read ofs=0 len=0 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 INFO: main: starting shard=0 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.622+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 INFO: main: starting shard=1 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 INFO: main: starting shard=2 2023-12-03T11:44:26.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.623+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 INFO: main: starting shard=3 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 INFO: main: starting shard=4 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.624+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 INFO: main: starting shard=5 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 INFO: main: starting shard=6 2023-12-03T11:44:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.625+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: starting shard=7 2023-12-03T11:44:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: starting shard=8 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 INFO: main: starting shard=9 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.626+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.627+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:26.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.627+0000 7fa7028c0800 20 INFO: main: starting shard=10 2023-12-03T11:44:26.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.627+0000 7fa7028c0800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:26.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.627+0000 7fa7028c0800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:44:26.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.627+0000 7fa7028c0800 20 INFO: main: done 2023-12-03T11:44:26.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.628+0000 7fa6c67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:26.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.630+0000 7fa7028c0800 20 remove_watcher() i=0 2023-12-03T11:44:26.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.630+0000 7fa7028c0800 2 removed watcher, disabling cache 2023-12-03T11:44:26.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.632+0000 7fa7028c0800 20 remove_watcher() i=1 2023-12-03T11:44:26.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.633+0000 7fa7028c0800 20 remove_watcher() i=2 2023-12-03T11:44:26.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.635+0000 7fa7028c0800 20 remove_watcher() i=3 2023-12-03T11:44:26.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.636+0000 7fa7028c0800 20 remove_watcher() i=4 2023-12-03T11:44:26.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.637+0000 7fa7028c0800 20 remove_watcher() i=5 2023-12-03T11:44:26.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.638+0000 7fa7028c0800 20 remove_watcher() i=6 2023-12-03T11:44:26.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:26.641+0000 7fa7028c0800 20 remove_watcher() i=7 2023-12-03T11:44:26.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:26.649 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:26.652 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:44:27.028 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:44:27.404 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check unlinked finds unlistable entries 2023-12-03T11:44:27.404 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.405 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:44:27.436 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:27.437 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:27.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.449+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 realm 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.451+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.452+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.452+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.453+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:27.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.453+0000 7f534a120800 20 searching for the correct realm 2023-12-03T11:44:27.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.463+0000 7f5345e7e640 0 -- 172.21.15.116:0/309170982 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x556629fbede0 msgr2=0x556629fc1210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.464+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 zone default found 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 4 Realm: () 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:27.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 zonegroup default 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.465+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.466+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.466+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.466+0000 7f534a120800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.466+0000 7f534a120800 20 started sync module instance, tier type = 2023-12-03T11:44:27.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.466+0000 7f534a120800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:27.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534567d640 0 -- 172.21.15.116:0/309170982 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x556629fc3e00 msgr2=0x556629fc61f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=0 2023-12-03T11:44:27.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=1 2023-12-03T11:44:27.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=2 2023-12-03T11:44:27.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=3 2023-12-03T11:44:27.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=4 2023-12-03T11:44:27.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.479+0000 7f534a120800 20 add_watcher() i=5 2023-12-03T11:44:27.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.483+0000 7f534a120800 20 add_watcher() i=6 2023-12-03T11:44:27.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.483+0000 7f534a120800 20 add_watcher() i=7 2023-12-03T11:44:27.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.483+0000 7f534a120800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:27.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.485+0000 7f530b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.486+0000 7f534a120800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:27.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.486+0000 7f534a120800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.491+0000 7f534a120800 5 note: GC not initialized 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.491+0000 7f527a7fc640 20 reqs_thread_entry: start 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.491+0000 7f534a120800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.491+0000 7f534a120800 20 Filter name: none 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f526bfff640 20 reqs_thread_entry: start 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:27.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.492+0000 7f534a120800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:44:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.493+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.494+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:27.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.494+0000 7f534a120800 1 NOTICE: finished shard 2 (0 entries found) 2023-12-03T11:44:27.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.494+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:27.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.494+0000 7f534a120800 1 NOTICE: finished shard 3 (0 entries found) 2023-12-03T11:44:27.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.494+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.495+0000 7f534a120800 1 NOTICE: finished shard 4 (0 entries found) 2023-12-03T11:44:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.495+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.495+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.496+0000 7f534a120800 1 NOTICE: finished shard 6 (0 entries found) 2023-12-03T11:44:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.496+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:27.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.496+0000 7f534a120800 1 NOTICE: finished shard 7 (0 entries found) 2023-12-03T11:44:27.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.496+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.497+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.497+0000 7f534a120800 1 NOTICE: finished shard 9 (0 entries found) 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.497+0000 7f534a120800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.498+0000 7f534a120800 1 NOTICE: finished shard 1 (1 entries found) 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.498+0000 7f534a120800 1 NOTICE: finished shard 5 (1 entries found) 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.498+0000 7f534a120800 1 NOTICE: finished shard 8 (1 entries found) 2023-12-03T11:44:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.498+0000 7f534a120800 1 NOTICE: finished shard 10 (1 entries found) 2023-12-03T11:44:27.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.498+0000 7f534a120800 1 NOTICE: finished all shards (4 entries found) 2023-12-03T11:44:27.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.499+0000 7f530b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.501+0000 7f534a120800 20 remove_watcher() i=0 2023-12-03T11:44:27.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.501+0000 7f534a120800 2 removed watcher, disabling cache 2023-12-03T11:44:27.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.502+0000 7f534a120800 20 remove_watcher() i=1 2023-12-03T11:44:27.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.504+0000 7f534a120800 20 remove_watcher() i=2 2023-12-03T11:44:27.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.505+0000 7f534a120800 20 remove_watcher() i=3 2023-12-03T11:44:27.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.506+0000 7f534a120800 20 remove_watcher() i=4 2023-12-03T11:44:27.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.507+0000 7f534a120800 20 remove_watcher() i=5 2023-12-03T11:44:27.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.510+0000 7f534a120800 20 remove_watcher() i=6 2023-12-03T11:44:27.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.511+0000 7f534a120800 20 remove_watcher() i=7 2023-12-03T11:44:27.517 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: unlinked entries are not listable 2023-12-03T11:44:27.517 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.519 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: GET returns 404 for unlinked entry keys that have no other versions 2023-12-03T11:44:27.523 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.527 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.571 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.613 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check unlinked fixes unlistable entries 2023-12-03T11:44:27.613 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.615 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:44:27.646 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:27.646 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:27.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.659+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.660+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.660+0000 7fabc6b25800 20 realm 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.660+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.661+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.661+0000 7fabc6b25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.661+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.661+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.661+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.662+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.662+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.662+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.662+0000 7fabc6b25800 20 searching for the correct realm 2023-12-03T11:44:27.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.672+0000 7fabc274d640 0 -- 172.21.15.116:0/3333946927 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557603224e70 msgr2=0x557603227260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.673+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 zone default found 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 4 Realm: () 2023-12-03T11:44:27.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 zonegroup default 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.674+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.675+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.675+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.675+0000 7fabc6b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.675+0000 7fabc6b25800 20 started sync module instance, tier type = 2023-12-03T11:44:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.675+0000 7fabc6b25800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:27.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.688+0000 7fabc2f4e640 0 -- 172.21.15.116:0/3333946927 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fabac0568b0 msgr2=0x7fabac0809f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=0 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=1 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=2 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=3 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=4 2023-12-03T11:44:27.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.693+0000 7fabc6b25800 20 add_watcher() i=5 2023-12-03T11:44:27.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.696+0000 7fabc6b25800 20 add_watcher() i=6 2023-12-03T11:44:27.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.696+0000 7fabc6b25800 20 add_watcher() i=7 2023-12-03T11:44:27.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.696+0000 7fabc6b25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.698+0000 7fab87fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.698+0000 7fabc6b25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.698+0000 7fabc6b25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:27.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 5 note: GC not initialized 2023-12-03T11:44:27.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7faaf6ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 20 Filter name: none 2023-12-03T11:44:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7faaf4ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.704+0000 7fabc6b25800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 1 NOTICE: finished shard 0 (0 entries removed) 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.705+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.706+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 1 NOTICE: finished shard 2 (0 entries removed) 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 1 NOTICE: finished shard 3 (0 entries removed) 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 1 NOTICE: finished shard 4 (0 entries removed) 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.707+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.708+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.708+0000 7fabc6b25800 1 NOTICE: finished shard 6 (0 entries removed) 2023-12-03T11:44:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.708+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.709+0000 7fabc6b25800 1 NOTICE: finished shard 7 (0 entries removed) 2023-12-03T11:44:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.709+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.709+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.710+0000 7fabc6b25800 1 NOTICE: finished shard 9 (0 entries removed) 2023-12-03T11:44:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.710+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.710+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.710+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabbc0cce00 obj=check-bucket:e state=0x5576032fe108 s->prefetch_data=0 2023-12-03T11:44:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.711+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.711+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabb4044040 obj=check-bucket:d state=0x55760333f178 s->prefetch_data=0 2023-12-03T11:44:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.711+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.711+0000 7fabc6b25800 20 get_obj_state: octx=0x557602fc10f0 obj=check-bucket:c state=0x55760333fcc8 s->prefetch_data=0 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.711+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state: octx=0x5576032b82e0 obj=check-bucket:f state=0x557603340818 s->prefetch_data=0 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to k6r2ev7nkzr46998ljje9rjnubmewd5x 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state_impl: setting s->olh_tag to 8g1lspsnw8c897w1vt7rv4itmeh64c0c 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=8g1lspsnw8c897w1vt7rv4itmeh64c0c 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to 0o61hookrho418kqip0p1l65te80v110 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state_impl: setting s->olh_tag to qcwglortyerjhxh4f9abb5723o33qwgk 2023-12-03T11:44:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=qcwglortyerjhxh4f9abb5723o33qwgk 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to hrng9tjk8k67fntmmj3tedxigho0a12y 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state_impl: setting s->olh_tag to 0l43r294cyuorrhy6qz6uknwncdb5646 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0l43r294cyuorrhy6qz6uknwncdb5646 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to aby0snptey1qt9yuyrkgqwl60rww8ck7 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 get_obj_state_impl: setting s->olh_tag to 3acx4rb8hyf131h29hbyp9lmv6t6y9r7 2023-12-03T11:44:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.712+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=3acx4rb8hyf131h29hbyp9lmv6t6y9r7 2023-12-03T11:44:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.716+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.716+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.716+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.716+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 olh_log_entry: epoch=3 op=3 key=d[IkdaAnYxiIxHDg2nQdaLP7MA0wh8rf.] 2023-12-03T11:44:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabb4044040 obj=check-bucket:_:IkdaAnYxiIxHDg2nQdaLP7MA0wh8rf._d state=0x55760335aeb8 s->prefetch_data=0 2023-12-03T11:44:27.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 10 manifest: total_size = 12 2023-12-03T11:44:27.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16280054323186757250 2023-12-03T11:44:27.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabb4044040 obj=check-bucket:_:IkdaAnYxiIxHDg2nQdaLP7MA0wh8rf._d state=0x55760335aeb8 s->prefetch_data=0 2023-12-03T11:44:27.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 prepare_atomic_modification: state is not atomic. state=0x55760335aeb8 2023-12-03T11:44:27.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.720+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 olh_log_entry: epoch=3 op=3 key=c[He02M6WnfuvKPabNWML1qpInkWwY9rw] 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: octx=0x557602fc10f0 obj=check-bucket:_:He02M6WnfuvKPabNWML1qpInkWwY9rw_c state=0x557603343388 s->prefetch_data=0 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 olh_log_entry: epoch=2 op=3 key=f[l.X8bOli7h2NzHmwSdL5FIWjBUwm46d] 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: octx=0x5576032b82e0 obj=check-bucket:_:l.X8bOli7h2NzHmwSdL5FIWjBUwm46d_f state=0x557603365bf8 s->prefetch_data=0 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 olh_log_entry: epoch=2 op=3 key=e[WUItWkbZKgo48sRuqo1aP-x96tP0KBA] 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabbc0cce00 obj=check-bucket:_:WUItWkbZKgo48sRuqo1aP-x96tP0KBA_e state=0x55760336c3e8 s->prefetch_data=0 2023-12-03T11:44:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 10 manifest: total_size = 9 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16256066844057806419 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: octx=0x5576032b82e0 obj=check-bucket:_:l.X8bOli7h2NzHmwSdL5FIWjBUwm46d_f state=0x557603365bf8 s->prefetch_data=0 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 prepare_atomic_modification: state is not atomic. state=0x557603365bf8 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 10 manifest: total_size = 22 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.687670627991377694 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 get_obj_state: octx=0x557602fc10f0 obj=check-bucket:_:He02M6WnfuvKPabNWML1qpInkWwY9rw_c state=0x557603343388 s->prefetch_data=0 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 prepare_atomic_modification: state is not atomic. state=0x557603343388 2023-12-03T11:44:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.722+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.723+0000 7fabc6b25800 10 manifest: total_size = 11 2023-12-03T11:44:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.723+0000 7fabc6b25800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5945218726263405482 2023-12-03T11:44:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.723+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabbc0cce00 obj=check-bucket:_:WUItWkbZKgo48sRuqo1aP-x96tP0KBA_e state=0x55760336c3e8 s->prefetch_data=0 2023-12-03T11:44:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.723+0000 7fabc6b25800 20 prepare_atomic_modification: state is not atomic. state=0x55760336c3e8 2023-12-03T11:44:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.723+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.730+0000 7fabc6b25800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:27.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.730+0000 7fabc6b25800 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2023-12-03T11:44:27.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.730+0000 7fabc6b25800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:27.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.730+0000 7fabc6b25800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:IkdaAnYxiIxHDg2nQdaLP7MA0wh8rf._.vmzzS7ls4YZ0AdlYAFP9cAZPdSYJ9-C_0 2023-12-03T11:44:27.734 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.732+0000 7faba3fff640 20 handle_completion(): completion ok for obj=d[IkdaAnYxiIxHDg2nQdaLP7MA0wh8rf.] 2023-12-03T11:44:27.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.735+0000 7fabc6b25800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:27.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.735+0000 7fabc6b25800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:27.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.735+0000 7fabc6b25800 20 RGWObjManifest::operator++(): result: ofs=9 stripe_ofs=9 part_ofs=0 rule->part_size=0 2023-12-03T11:44:27.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.735+0000 7fabc6b25800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:27.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.735+0000 7fabc6b25800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:l.X8bOli7h2NzHmwSdL5FIWjBUwm46d_.2FLvgV6VNMoi5AP64O8zEbjVj-Z_BVw_0 2023-12-03T11:44:27.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.738+0000 7fabb8f4a640 20 handle_completion(): completion ok for obj=f[l.X8bOli7h2NzHmwSdL5FIWjBUwm46d] 2023-12-03T11:44:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.740+0000 7fabc6b25800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.741+0000 7fabc6b25800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.741+0000 7fabc6b25800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:44:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.741+0000 7fabc6b25800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.741+0000 7fabc6b25800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:WUItWkbZKgo48sRuqo1aP-x96tP0KBA_.Mt9mXh5Tg2pigRIYiW229rTSpHOxFiF_0 2023-12-03T11:44:27.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.742+0000 7fabb8f4a640 20 handle_completion(): completion ok for obj=e[WUItWkbZKgo48sRuqo1aP-x96tP0KBA] 2023-12-03T11:44:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.746+0000 7fabc6b25800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.746+0000 7fabc6b25800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.746+0000 7fabc6b25800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:44:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.746+0000 7fabc6b25800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.746+0000 7fabc6b25800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:He02M6WnfuvKPabNWML1qpInkWwY9rw_.fLJ_VahovO_JFPktZXpa0jnkTKcR4gp_0 2023-12-03T11:44:27.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.754+0000 7faba3fff640 20 handle_completion(): completion ok for obj=c[He02M6WnfuvKPabNWML1qpInkWwY9rw] 2023-12-03T11:44:27.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 get_obj_state: octx=0x5576032b82e0 obj=check-bucket:f state=0x557603340818 s->prefetch_data=0 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=3acx4rb8hyf131h29hbyp9lmv6t6y9r7 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 get_obj_state: octx=0x7fabbc0cce00 obj=check-bucket:e state=0x5576032fe108 s->prefetch_data=0 2023-12-03T11:44:27.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.755+0000 7fabc6b25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=8g1lspsnw8c897w1vt7rv4itmeh64c0c 2023-12-03T11:44:27.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.756+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.756+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.756+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.756+0000 7fabc6b25800 1 NOTICE: finished shard 5 (1 entries removed) 2023-12-03T11:44:27.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.762+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.762+0000 7fabc6b25800 1 NOTICE: finished shard 8 (1 entries removed) 2023-12-03T11:44:27.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.764+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.764+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.766+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.766+0000 7fabc6b25800 1 NOTICE: finished shard 1 (1 entries removed) 2023-12-03T11:44:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.769+0000 7fabc6b25800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.769+0000 7fabc6b25800 1 NOTICE: finished shard 10 (1 entries removed) 2023-12-03T11:44:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.769+0000 7fabc6b25800 1 NOTICE: finished all shards (4 entries removed) 2023-12-03T11:44:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.770+0000 7fab87fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.773+0000 7fabc6b25800 20 remove_watcher() i=0 2023-12-03T11:44:27.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.773+0000 7fabc6b25800 2 removed watcher, disabling cache 2023-12-03T11:44:27.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.775+0000 7fabc6b25800 20 remove_watcher() i=1 2023-12-03T11:44:27.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.777+0000 7fabc6b25800 20 remove_watcher() i=2 2023-12-03T11:44:27.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.778+0000 7fabc6b25800 20 remove_watcher() i=3 2023-12-03T11:44:27.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.780+0000 7fabc6b25800 20 remove_watcher() i=4 2023-12-03T11:44:27.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.781+0000 7fabc6b25800 20 remove_watcher() i=5 2023-12-03T11:44:27.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.782+0000 7fabc6b25800 20 remove_watcher() i=6 2023-12-03T11:44:27.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.783+0000 7fabc6b25800 20 remove_watcher() i=7 2023-12-03T11:44:27.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.791 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.794 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.796 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.799 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.800 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check unlinked does not affect normal entries 2023-12-03T11:44:27.800 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.801 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.846 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.849 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.851 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.854 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.855 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check unlinked does not find new unlistable entries after fix 2023-12-03T11:44:27.856 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:27.857 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:44:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:27.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.901+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 realm 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.903+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.904+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.904+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.904+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:27.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.904+0000 7fb6c9211800 20 searching for the correct realm 2023-12-03T11:44:27.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.914+0000 7fb6c47b1640 0 -- 172.21.15.116:0/3068827713 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55d2f5a8a5d0 msgr2=0x55d2f5a8c9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.915+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:27.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:27.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 zone default found 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 4 Realm: () 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 zonegroup default 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.916+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.917+0000 7fb6c9211800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.917+0000 7fb6c9211800 20 started sync module instance, tier type = 2023-12-03T11:44:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.917+0000 7fb6c9211800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:27.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.928+0000 7fb6bbfb0640 0 -- 172.21.15.116:0/3068827713 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55d2f5a8f540 msgr2=0x55d2f5a91930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:27.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=0 2023-12-03T11:44:27.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=1 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=2 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=3 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=4 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=5 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.932+0000 7fb6c9211800 20 add_watcher() i=6 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.933+0000 7fb6c9211800 20 add_watcher() i=7 2023-12-03T11:44:27.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.933+0000 7fb6c9211800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:27.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.935+0000 7fb68a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.935+0000 7fb6c9211800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:27.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.935+0000 7fb6c9211800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:27.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 5 note: GC not initialized 2023-12-03T11:44:27.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb5f97fa640 20 reqs_thread_entry: start 2023-12-03T11:44:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 20 Filter name: none 2023-12-03T11:44:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb5f2ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.941+0000 7fb6c9211800 20 rados->read ofs=0 len=0 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.942+0000 7fb6c9211800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.943+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.943+0000 7fb6c9211800 1 NOTICE: finished shard 1 (0 entries found) 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.943+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.943+0000 7fb6c9211800 1 NOTICE: finished shard 2 (0 entries found) 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.943+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.944+0000 7fb6c9211800 1 NOTICE: finished shard 3 (0 entries found) 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.944+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.944+0000 7fb6c9211800 1 NOTICE: finished shard 4 (0 entries found) 2023-12-03T11:44:27.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.944+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 1 NOTICE: finished shard 5 (0 entries found) 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 1 NOTICE: finished shard 6 (0 entries found) 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 1 NOTICE: finished shard 7 (0 entries found) 2023-12-03T11:44:27.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.945+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.946+0000 7fb6c9211800 1 NOTICE: finished shard 8 (0 entries found) 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.946+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.946+0000 7fb6c9211800 1 NOTICE: finished shard 9 (0 entries found) 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.946+0000 7fb6c9211800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.947+0000 7fb6c9211800 1 NOTICE: finished shard 10 (0 entries found) 2023-12-03T11:44:27.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.947+0000 7fb6c9211800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:44:27.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.947+0000 7fb68a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.950+0000 7fb6c9211800 20 remove_watcher() i=0 2023-12-03T11:44:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.950+0000 7fb6c9211800 2 removed watcher, disabling cache 2023-12-03T11:44:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.951+0000 7fb6c9211800 20 remove_watcher() i=1 2023-12-03T11:44:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.952+0000 7fb6c9211800 20 remove_watcher() i=2 2023-12-03T11:44:27.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.954+0000 7fb6c9211800 20 remove_watcher() i=3 2023-12-03T11:44:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.955+0000 7fb6c9211800 20 remove_watcher() i=4 2023-12-03T11:44:27.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.956+0000 7fb6c9211800 20 remove_watcher() i=5 2023-12-03T11:44:27.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.957+0000 7fb6c9211800 20 remove_watcher() i=6 2023-12-03T11:44:27.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:27.958+0000 7fb6c9211800 20 remove_watcher() i=7 2023-12-03T11:44:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.967 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:27.981 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:27.996 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:44:28.024 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.024 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.036+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 realm 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.038+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.039+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.039+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.039+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.039+0000 7f67a180f800 20 searching for the correct realm 2023-12-03T11:44:28.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.049+0000 7f679cf47640 0 -- 172.21.15.116:0/1442477271 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55ed9c919b50 msgr2=0x55ed9c91bf60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.050+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 zone default found 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 4 Realm: () 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 zonegroup default 2023-12-03T11:44:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 started sync module instance, tier type = 2023-12-03T11:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.051+0000 7f67a180f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.063+0000 7f679c746640 0 -- 172.21.15.116:0/1442477271 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55ed9c91eba0 msgr2=0x55ed9c920f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=0 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=1 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=2 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=3 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=4 2023-12-03T11:44:28.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.064+0000 7f67a180f800 20 add_watcher() i=5 2023-12-03T11:44:28.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.065+0000 7f67a180f800 20 add_watcher() i=6 2023-12-03T11:44:28.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.065+0000 7f67a180f800 20 add_watcher() i=7 2023-12-03T11:44:28.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.065+0000 7f67a180f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.067+0000 7f6762fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.067+0000 7f67a180f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.067+0000 7f67a180f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f67a180f800 5 note: GC not initialized 2023-12-03T11:44:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f66d1ffb640 20 reqs_thread_entry: start 2023-12-03T11:44:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f67a180f800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f67a180f800 20 Filter name: none 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f66c37fe640 20 reqs_thread_entry: start 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.073+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 INFO: main: starting shard=0 2023-12-03T11:44:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 INFO: main: starting shard=1 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.074+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 INFO: main: starting shard=2 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 INFO: main: starting shard=3 2023-12-03T11:44:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.075+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 INFO: main: starting shard=4 2023-12-03T11:44:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 INFO: main: starting shard=5 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.076+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 INFO: main: starting shard=6 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 INFO: main: starting shard=7 2023-12-03T11:44:28.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.077+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 INFO: main: starting shard=8 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 INFO: main: starting shard=9 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.078+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.079+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.079+0000 7f67a180f800 20 INFO: main: starting shard=10 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.079+0000 7f67a180f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.079+0000 7f67a180f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.079+0000 7f67a180f800 20 INFO: main: done 2023-12-03T11:44:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.080+0000 7f6762fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.082+0000 7f67a180f800 20 remove_watcher() i=0 2023-12-03T11:44:28.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.082+0000 7f67a180f800 2 removed watcher, disabling cache 2023-12-03T11:44:28.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.083+0000 7f67a180f800 20 remove_watcher() i=1 2023-12-03T11:44:28.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.084+0000 7f67a180f800 20 remove_watcher() i=2 2023-12-03T11:44:28.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.086+0000 7f67a180f800 20 remove_watcher() i=3 2023-12-03T11:44:28.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.087+0000 7f67a180f800 20 remove_watcher() i=4 2023-12-03T11:44:28.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.088+0000 7f67a180f800 20 remove_watcher() i=5 2023-12-03T11:44:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.091+0000 7f67a180f800 20 remove_watcher() i=6 2023-12-03T11:44:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.092+0000 7f67a180f800 20 remove_watcher() i=7 2023-12-03T11:44:28.099 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:44:28.128 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.128 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.140+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 20 realm 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.141+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.142+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.142+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.142+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.142+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.143+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.143+0000 7f962251e800 20 searching for the correct realm 2023-12-03T11:44:28.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.152+0000 7f9620025640 0 -- 172.21.15.116:0/2883317381 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5641cdf06f70 msgr2=0x5641cdf09360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.153+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 zone default found 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 4 Realm: () 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 zonegroup default 2023-12-03T11:44:28.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.154+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.155+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.155+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.155+0000 7f962251e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.155+0000 7f962251e800 20 started sync module instance, tier type = 2023-12-03T11:44:28.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.155+0000 7f962251e800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.169+0000 7f961f824640 0 -- 172.21.15.116:0/2883317381 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5641cdf0bec0 msgr2=0x5641cdf0e2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=0 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=1 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=2 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=3 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=4 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=5 2023-12-03T11:44:28.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.177+0000 7f962251e800 20 add_watcher() i=6 2023-12-03T11:44:28.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.180+0000 7f962251e800 20 add_watcher() i=7 2023-12-03T11:44:28.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.180+0000 7f962251e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.182+0000 7f95e67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.182+0000 7f962251e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.182+0000 7f962251e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 5 note: GC not initialized 2023-12-03T11:44:28.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f95557fa640 20 reqs_thread_entry: start 2023-12-03T11:44:28.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 20 Filter name: none 2023-12-03T11:44:28.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f954effd640 20 reqs_thread_entry: start 2023-12-03T11:44:28.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.188+0000 7f962251e800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 INFO: main: starting shard=0 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 INFO: main: starting shard=1 2023-12-03T11:44:28.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.189+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.190+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.190+0000 7f962251e800 20 INFO: main: starting shard=2 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.190+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: starting shard=3 2023-12-03T11:44:28.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: starting shard=4 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 INFO: main: starting shard=5 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.191+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 INFO: main: starting shard=6 2023-12-03T11:44:28.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 INFO: main: starting shard=7 2023-12-03T11:44:28.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.192+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: starting shard=8 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: starting shard=9 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 INFO: main: starting shard=10 2023-12-03T11:44:28.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.193+0000 7f962251e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.194+0000 7f962251e800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.194+0000 7f962251e800 20 INFO: main: done 2023-12-03T11:44:28.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.194+0000 7f95e67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.197+0000 7f962251e800 20 remove_watcher() i=0 2023-12-03T11:44:28.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.197+0000 7f962251e800 2 removed watcher, disabling cache 2023-12-03T11:44:28.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.198+0000 7f962251e800 20 remove_watcher() i=1 2023-12-03T11:44:28.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.199+0000 7f962251e800 20 remove_watcher() i=2 2023-12-03T11:44:28.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.201+0000 7f962251e800 20 remove_watcher() i=3 2023-12-03T11:44:28.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.203+0000 7f962251e800 20 remove_watcher() i=4 2023-12-03T11:44:28.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.204+0000 7f962251e800 20 remove_watcher() i=5 2023-12-03T11:44:28.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.205+0000 7f962251e800 20 remove_watcher() i=6 2023-12-03T11:44:28.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.206+0000 7f962251e800 20 remove_watcher() i=7 2023-12-03T11:44:28.211 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check olh finds leftover OLH entries 2023-12-03T11:44:28.211 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:28.213 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-12-03T11:44:28.241 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.242 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.254+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 realm 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.256+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.257+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.257+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.257+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.257+0000 7ff8b9a0f800 20 searching for the correct realm 2023-12-03T11:44:28.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.266+0000 7ff8b750f640 0 -- 172.21.15.116:0/3581608120 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x564d00473e10 msgr2=0x564d00476200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.267+0000 7ff8b9a0f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.267+0000 7ff8b9a0f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.267+0000 7ff8b9a0f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.267+0000 7ff8b9a0f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.267+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 zone default found 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 4 Realm: () 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 zonegroup default 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.268+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 started sync module instance, tier type = 2023-12-03T11:44:28.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.269+0000 7ff8b9a0f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.279+0000 7ff8b750f640 0 -- 172.21.15.116:0/3581608120 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7ff8b0060520 msgr2=0x7ff8b00599e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=0 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=1 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=2 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=3 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=4 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=5 2023-12-03T11:44:28.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.282+0000 7ff8b9a0f800 20 add_watcher() i=6 2023-12-03T11:44:28.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.284+0000 7ff8b9a0f800 20 add_watcher() i=7 2023-12-03T11:44:28.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.284+0000 7ff8b9a0f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.286+0000 7ff87d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.286+0000 7ff8b9a0f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.286+0000 7ff8b9a0f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff8b9a0f800 5 note: GC not initialized 2023-12-03T11:44:28.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff7ebfff640 20 reqs_thread_entry: start 2023-12-03T11:44:28.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff8b9a0f800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff8b9a0f800 20 Filter name: none 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff7e9ffb640 20 reqs_thread_entry: start 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.292+0000 7ff8b9a0f800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:28.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.293+0000 7ff8b9a0f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:28.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:28.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:28.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.294+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 1 NOTICE: finished shard 1 (0 entries found) 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 1 NOTICE: finished shard 2 (0 entries found) 2023-12-03T11:44:28.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 1 NOTICE: finished shard 3 (0 entries found) 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.295+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.296+0000 7ff8b9a0f800 1 NOTICE: finished shard 4 (0 entries found) 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.296+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.296+0000 7ff8b9a0f800 1 NOTICE: finished shard 5 (1 entries found) 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.296+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 1 NOTICE: finished shard 6 (0 entries found) 2023-12-03T11:44:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 1 NOTICE: finished shard 7 (0 entries found) 2023-12-03T11:44:28.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 1 NOTICE: finished shard 8 (1 entries found) 2023-12-03T11:44:28.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.297+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.298+0000 7ff8b9a0f800 1 NOTICE: finished shard 9 (0 entries found) 2023-12-03T11:44:28.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.298+0000 7ff8b9a0f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.298+0000 7ff8b9a0f800 1 NOTICE: finished shard 10 (0 entries found) 2023-12-03T11:44:28.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.298+0000 7ff8b9a0f800 1 NOTICE: finished all shards (2 entries found) 2023-12-03T11:44:28.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.299+0000 7ff87d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.301+0000 7ff8b9a0f800 20 remove_watcher() i=0 2023-12-03T11:44:28.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.301+0000 7ff8b9a0f800 2 removed watcher, disabling cache 2023-12-03T11:44:28.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.303+0000 7ff8b9a0f800 20 remove_watcher() i=1 2023-12-03T11:44:28.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.304+0000 7ff8b9a0f800 20 remove_watcher() i=2 2023-12-03T11:44:28.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.306+0000 7ff8b9a0f800 20 remove_watcher() i=3 2023-12-03T11:44:28.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.307+0000 7ff8b9a0f800 20 remove_watcher() i=4 2023-12-03T11:44:28.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.308+0000 7ff8b9a0f800 20 remove_watcher() i=5 2023-12-03T11:44:28.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.309+0000 7ff8b9a0f800 20 remove_watcher() i=6 2023-12-03T11:44:28.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.310+0000 7ff8b9a0f800 20 remove_watcher() i=7 2023-12-03T11:44:28.316 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check olh fixes leftover OLH entries 2023-12-03T11:44:28.316 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:28.317 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check olh --bucket check-bucket --fix --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:44:28.349 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.349 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.361+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 realm 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.362+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.363+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.363+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.363+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.363+0000 7fc9b972b800 20 searching for the correct realm 2023-12-03T11:44:28.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.372+0000 7fc9b5432640 0 -- 172.21.15.116:0/3609470084 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55b67ef7c630 msgr2=0x55b67ef7ea80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.374+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 zone default found 2023-12-03T11:44:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 4 Realm: () 2023-12-03T11:44:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 zonegroup default 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.375+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.376+0000 7fc9b972b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.376+0000 7fc9b972b800 20 started sync module instance, tier type = 2023-12-03T11:44:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.376+0000 7fc9b972b800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.384+0000 7fc9b5c33640 0 -- 172.21.15.116:0/3609470084 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fc9a0037950 msgr2=0x7fc9a00363d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:28.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=0 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=1 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=2 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=3 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=4 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=5 2023-12-03T11:44:28.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.387+0000 7fc9b972b800 20 add_watcher() i=6 2023-12-03T11:44:28.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.389+0000 7fc9b972b800 20 add_watcher() i=7 2023-12-03T11:44:28.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.389+0000 7fc9b972b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.390+0000 7fc97afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.391+0000 7fc9b972b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.391+0000 7fc9b972b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.396+0000 7fc9b972b800 5 note: GC not initialized 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.396+0000 7fc8e9ffb640 20 reqs_thread_entry: start 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.396+0000 7fc9b972b800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 20 Filter name: none 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc8db7fe640 20 reqs_thread_entry: start 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.397+0000 7fc9b972b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 1 NOTICE: finished shard 0 (0 entries removed) 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.398+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 1 NOTICE: finished shard 1 (0 entries removed) 2023-12-03T11:44:28.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 1 NOTICE: finished shard 2 (0 entries removed) 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 1 NOTICE: finished shard 3 (0 entries removed) 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.399+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.400+0000 7fc9b972b800 1 NOTICE: finished shard 4 (0 entries removed) 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.400+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.400+0000 7fc9b972b800 20 get_obj_state: octx=0x7fc9a8009f80 obj=check-bucket:d state=0x55b67f02f388 s->prefetch_data=0 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.400+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 1 NOTICE: finished shard 6 (0 entries removed) 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 1 NOTICE: finished shard 7 (0 entries removed) 2023-12-03T11:44:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 20 get_obj_state: octx=0x55b67ed1b0f0 obj=check-bucket:c state=0x55b67f053448 s->prefetch_data=0 2023-12-03T11:44:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.401+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 1 NOTICE: finished shard 9 (0 entries removed) 2023-12-03T11:44:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 1 NOTICE: finished shard 10 (0 entries removed) 2023-12-03T11:44:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 get_obj_state: setting s->obj_tag to 0o61hookrho418kqip0p1l65te80v110 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 get_obj_state_impl: setting s->olh_tag to qcwglortyerjhxh4f9abb5723o33qwgk 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 get_obj_state: setting s->obj_tag to hrng9tjk8k67fntmmj3tedxigho0a12y 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 get_obj_state_impl: setting s->olh_tag to 0l43r294cyuorrhy6qz6uknwncdb5646 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 olh_log_entry: epoch=4 op=2 key=d[] 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.402+0000 7fc9b972b800 20 olh_log_entry: epoch=4 op=3 key=d[DK8SJY8Hr6zsa4W8CSI1eg1D3zp6MJ5] 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.403+0000 7fc9b972b800 20 get_obj_state: octx=0x55b67f02e7f0 obj=check-bucket:_:DK8SJY8Hr6zsa4W8CSI1eg1D3zp6MJ5_d state=0x55b67f059968 s->prefetch_data=0 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.403+0000 7fc9b972b800 20 olh_log_entry: epoch=4 op=2 key=c[] 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.403+0000 7fc9b972b800 20 olh_log_entry: epoch=4 op=3 key=c[66icS44Wrnc16T31IVtXZdaQWy03zey] 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.403+0000 7fc9b972b800 20 get_obj_state: octx=0x55b67f0525f0 obj=check-bucket:_:66icS44Wrnc16T31IVtXZdaQWy03zey_c state=0x55b67f061818 s->prefetch_data=0 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.404+0000 7fc9b972b800 20 get_obj_state: octx=0x55b67f02e7f0 obj=check-bucket:d state=0x55b67f074be8 s->prefetch_data=0 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.404+0000 7fc9b972b800 20 get_obj_state: setting s->obj_tag to 0o61hookrho418kqip0p1l65te80v110 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.404+0000 7fc9b972b800 20 get_obj_state_impl: setting s->olh_tag to qcwglortyerjhxh4f9abb5723o33qwgk 2023-12-03T11:44:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.404+0000 7fc9b972b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=qcwglortyerjhxh4f9abb5723o33qwgk 2023-12-03T11:44:28.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.406+0000 7fc9b972b800 20 get_obj_state: octx=0x55b67f0525f0 obj=check-bucket:c state=0x55b67f0607c8 s->prefetch_data=0 2023-12-03T11:44:28.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.406+0000 7fc9b972b800 20 get_obj_state: setting s->obj_tag to hrng9tjk8k67fntmmj3tedxigho0a12y 2023-12-03T11:44:28.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.406+0000 7fc9b972b800 20 get_obj_state_impl: setting s->olh_tag to 0l43r294cyuorrhy6qz6uknwncdb5646 2023-12-03T11:44:28.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.406+0000 7fc9b972b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0l43r294cyuorrhy6qz6uknwncdb5646 2023-12-03T11:44:28.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.413+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.416+0000 7fc9b972b800 1 NOTICE: finished shard 5 (1 entries removed) 2023-12-03T11:44:28.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.416+0000 7fc9b972b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.417+0000 7fc9b972b800 1 NOTICE: finished shard 8 (1 entries removed) 2023-12-03T11:44:28.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.418+0000 7fc9b972b800 1 NOTICE: finished all shards (2 entries removed) 2023-12-03T11:44:28.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.418+0000 7fc97afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.420+0000 7fc9b972b800 20 remove_watcher() i=0 2023-12-03T11:44:28.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.420+0000 7fc9b972b800 2 removed watcher, disabling cache 2023-12-03T11:44:28.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.423+0000 7fc9b972b800 20 remove_watcher() i=1 2023-12-03T11:44:28.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.424+0000 7fc9b972b800 20 remove_watcher() i=2 2023-12-03T11:44:28.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.425+0000 7fc9b972b800 20 remove_watcher() i=3 2023-12-03T11:44:28.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.426+0000 7fc9b972b800 20 remove_watcher() i=4 2023-12-03T11:44:28.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.427+0000 7fc9b972b800 20 remove_watcher() i=5 2023-12-03T11:44:28.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.428+0000 7fc9b972b800 20 remove_watcher() i=6 2023-12-03T11:44:28.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.429+0000 7fc9b972b800 20 remove_watcher() i=7 2023-12-03T11:44:28.436 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:44:28.465 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.465 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.477+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 realm 2023-12-03T11:44:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.479+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.480+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.480+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.480+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.480+0000 7fa108633800 20 searching for the correct realm 2023-12-03T11:44:28.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.489+0000 7fa105939640 0 -- 172.21.15.116:0/1275455907 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x558886ba57f0 msgr2=0x558886ba7be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.490+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 zone default found 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 4 Realm: () 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 zonegroup default 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.491+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.492+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.492+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.492+0000 7fa108633800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.492+0000 7fa108633800 20 started sync module instance, tier type = 2023-12-03T11:44:28.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.492+0000 7fa108633800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.498+0000 7fa10693b640 0 -- 172.21.15.116:0/1275455907 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7fa0fc069020 msgr2=0x7fa0fc097c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=0 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=1 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=2 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=3 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=4 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=5 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.507+0000 7fa108633800 20 add_watcher() i=6 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.509+0000 7fa108633800 20 add_watcher() i=7 2023-12-03T11:44:28.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.509+0000 7fa108633800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.512+0000 7fa0cbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.512+0000 7fa108633800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.512+0000 7fa108633800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa108633800 5 note: GC not initialized 2023-12-03T11:44:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa03affd640 20 reqs_thread_entry: start 2023-12-03T11:44:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa108633800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa108633800 20 Filter name: none 2023-12-03T11:44:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa038ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.518+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 INFO: main: starting shard=0 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.519+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: starting shard=1 2023-12-03T11:44:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: starting shard=2 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 INFO: main: starting shard=3 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.520+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 INFO: main: starting shard=4 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 INFO: main: starting shard=5 2023-12-03T11:44:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.521+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 INFO: main: starting shard=6 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 INFO: main: starting shard=7 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.522+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 INFO: main: starting shard=8 2023-12-03T11:44:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 INFO: main: starting shard=9 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.523+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.524+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.524+0000 7fa108633800 20 INFO: main: starting shard=10 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.524+0000 7fa108633800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.524+0000 7fa108633800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.524+0000 7fa108633800 20 INFO: main: done 2023-12-03T11:44:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.525+0000 7fa0cbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.527+0000 7fa108633800 20 remove_watcher() i=0 2023-12-03T11:44:28.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.527+0000 7fa108633800 2 removed watcher, disabling cache 2023-12-03T11:44:28.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.528+0000 7fa108633800 20 remove_watcher() i=1 2023-12-03T11:44:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.529+0000 7fa108633800 20 remove_watcher() i=2 2023-12-03T11:44:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.530+0000 7fa108633800 20 remove_watcher() i=3 2023-12-03T11:44:28.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.535+0000 7fa108633800 20 remove_watcher() i=4 2023-12-03T11:44:28.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.536+0000 7fa108633800 20 remove_watcher() i=5 2023-12-03T11:44:28.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.538+0000 7fa108633800 20 remove_watcher() i=6 2023-12-03T11:44:28.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.539+0000 7fa108633800 20 remove_watcher() i=7 2023-12-03T11:44:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:44:28.575 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.575 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.588+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 realm 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.590+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.591+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.591+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.592+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.592+0000 7f24c2725800 20 searching for the correct realm 2023-12-03T11:44:28.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.601+0000 7f24be354640 0 -- 172.21.15.116:0/3459163207 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5644ad431950 msgr2=0x5644ad433d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.602+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 zone default found 2023-12-03T11:44:28.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 4 Realm: () 2023-12-03T11:44:28.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 zonegroup default 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.603+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.604+0000 7f24c2725800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.604+0000 7f24c2725800 20 started sync module instance, tier type = 2023-12-03T11:44:28.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.604+0000 7f24c2725800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.612+0000 7f24be354640 0 -- 172.21.15.116:0/3459163207 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f24b8067070 msgr2=0x7f24b8099460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=0 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=1 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=2 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=3 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=4 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=5 2023-12-03T11:44:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.623+0000 7f24c2725800 20 add_watcher() i=6 2023-12-03T11:44:28.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.626+0000 7f24c2725800 20 add_watcher() i=7 2023-12-03T11:44:28.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.626+0000 7f24c2725800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.627+0000 7f2483fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.628+0000 7f24c2725800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.628+0000 7f24c2725800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.633+0000 7f24c2725800 5 note: GC not initialized 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.633+0000 7f23f2ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.633+0000 7f24c2725800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.633+0000 7f24c2725800 20 Filter name: none 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.633+0000 7f23f0ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:28.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 INFO: main: starting shard=0 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.634+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 INFO: main: starting shard=1 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 INFO: main: starting shard=2 2023-12-03T11:44:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.635+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: starting shard=3 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: starting shard=4 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 INFO: main: starting shard=5 2023-12-03T11:44:28.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.636+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 INFO: main: starting shard=6 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 INFO: main: starting shard=7 2023-12-03T11:44:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.637+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: starting shard=8 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: starting shard=9 2023-12-03T11:44:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 INFO: main: starting shard=10 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.638+0000 7f24c2725800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.639+0000 7f24c2725800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.639+0000 7f24c2725800 20 INFO: main: done 2023-12-03T11:44:28.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.639+0000 7f2483fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.642+0000 7f24c2725800 20 remove_watcher() i=0 2023-12-03T11:44:28.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.642+0000 7f24c2725800 2 removed watcher, disabling cache 2023-12-03T11:44:28.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.643+0000 7f24c2725800 20 remove_watcher() i=1 2023-12-03T11:44:28.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.644+0000 7f24c2725800 20 remove_watcher() i=2 2023-12-03T11:44:28.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.645+0000 7f24c2725800 20 remove_watcher() i=3 2023-12-03T11:44:28.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.646+0000 7f24c2725800 20 remove_watcher() i=4 2023-12-03T11:44:28.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.647+0000 7f24c2725800 20 remove_watcher() i=5 2023-12-03T11:44:28.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.648+0000 7f24c2725800 20 remove_watcher() i=6 2023-12-03T11:44:28.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.649+0000 7f24c2725800 20 remove_watcher() i=7 2023-12-03T11:44:28.655 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check olh does not find new leftover OLH entries after fix 2023-12-03T11:44:28.655 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-12-03T11:44:28.685 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:28.685 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:28.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.698+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 20 realm 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.699+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.700+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.700+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.700+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.700+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.701+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:28.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.701+0000 7f9a7b92b800 20 searching for the correct realm 2023-12-03T11:44:28.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.710+0000 7f9a77632640 0 -- 172.21.15.116:0/2196520486 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f9a700571a0 msgr2=0x7f9a70056c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:28.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.711+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 zone default found 2023-12-03T11:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 4 Realm: () 2023-12-03T11:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 zonegroup default 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.712+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.713+0000 7f9a7b92b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.713+0000 7f9a7b92b800 20 started sync module instance, tier type = 2023-12-03T11:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.713+0000 7f9a7b92b800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:28.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.720+0000 7f9a77e33640 0 -- 172.21.15.116:0/2196520486 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9a600663f0 msgr2=0x7f9a60066850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=0 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=1 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=2 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=3 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=4 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=5 2023-12-03T11:44:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.725+0000 7f9a7b92b800 20 add_watcher() i=6 2023-12-03T11:44:28.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.728+0000 7f9a7b92b800 20 add_watcher() i=7 2023-12-03T11:44:28.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.728+0000 7f9a7b92b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:28.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.729+0000 7f9a3d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.730+0000 7f9a7b92b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:28.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.730+0000 7f9a7b92b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.735+0000 7f9a7b92b800 5 note: GC not initialized 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.735+0000 7f99abfff640 20 reqs_thread_entry: start 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 Filter name: none 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f99a9ffb640 20 reqs_thread_entry: start 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.736+0000 7f9a7b92b800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:44:28.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.737+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 1 NOTICE: finished shard 1 (0 entries found) 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 1 NOTICE: finished shard 2 (0 entries found) 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 1 NOTICE: finished shard 3 (0 entries found) 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.738+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.739+0000 7f9a7b92b800 1 NOTICE: finished shard 4 (0 entries found) 2023-12-03T11:44:28.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.739+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:28.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.739+0000 7f9a7b92b800 1 NOTICE: finished shard 5 (0 entries found) 2023-12-03T11:44:28.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.739+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:28.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.740+0000 7f9a7b92b800 1 NOTICE: finished shard 6 (0 entries found) 2023-12-03T11:44:28.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.740+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.740+0000 7f9a7b92b800 1 NOTICE: finished shard 7 (0 entries found) 2023-12-03T11:44:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.740+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 1 NOTICE: finished shard 8 (0 entries found) 2023-12-03T11:44:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 1 NOTICE: finished shard 9 (0 entries found) 2023-12-03T11:44:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 1 NOTICE: finished shard 10 (0 entries found) 2023-12-03T11:44:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.741+0000 7f9a7b92b800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:44:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.742+0000 7f9a3d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:28.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.744+0000 7f9a7b92b800 20 remove_watcher() i=0 2023-12-03T11:44:28.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.744+0000 7f9a7b92b800 2 removed watcher, disabling cache 2023-12-03T11:44:28.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.746+0000 7f9a7b92b800 20 remove_watcher() i=1 2023-12-03T11:44:28.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.752+0000 7f9a7b92b800 20 remove_watcher() i=2 2023-12-03T11:44:28.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.752+0000 7f9a7b92b800 20 remove_watcher() i=3 2023-12-03T11:44:28.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.754+0000 7f9a7b92b800 20 remove_watcher() i=4 2023-12-03T11:44:28.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.755+0000 7f9a7b92b800 20 remove_watcher() i=5 2023-12-03T11:44:28.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.756+0000 7f9a7b92b800 20 remove_watcher() i=6 2023-12-03T11:44:28.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:28.757+0000 7f9a7b92b800 20 remove_watcher() i=7 2023-12-03T11:44:28.763 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: verify that bucket check fixes do not affect null version objects 2023-12-03T11:44:28.764 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:28.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.765 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.769 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.771 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: bucket check stats are correct in the presence of unlinked entries 2023-12-03T11:44:28.815 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:44:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.815 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.860 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.903 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.919 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.921 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.946 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.949 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.979 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:28.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:28.982 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:29.004 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:29.006 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:29.024 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:29.027 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:29.048 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:44:29.426 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:44:30.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:30.803 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:30.856 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:30.857 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:44:30.890 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:30.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.906+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 20 realm 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.908+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.909+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.909+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.910+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:30.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.910+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.910+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:30.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.910+0000 7f1c7f93a800 20 searching for the correct realm 2023-12-03T11:44:30.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.919+0000 7f1c72e05640 0 -- 172.21.15.116:0/1759043774 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x55b858cdccd0 msgr2=0x55b858cdf0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.923+0000 7f1c7f93a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.923+0000 7f1c7f93a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.923+0000 7f1c7f93a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.923+0000 7f1c7f93a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.923+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 zone default found 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 4 Realm: () 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 zonegroup default 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.924+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.925+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.925+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.925+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.925+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.925+0000 7f1c72e05640 0 -- 172.21.15.116:0/1759043774 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55b858ce5b50 msgr2=0x55b858ce7f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.926+0000 7f1c7f93a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.926+0000 7f1c7f93a800 20 started sync module instance, tier type = 2023-12-03T11:44:30.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.926+0000 7f1c7f93a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:30.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=0 2023-12-03T11:44:30.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=1 2023-12-03T11:44:30.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=2 2023-12-03T11:44:30.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=3 2023-12-03T11:44:30.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=4 2023-12-03T11:44:30.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=5 2023-12-03T11:44:30.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.940+0000 7f1c7f93a800 20 add_watcher() i=6 2023-12-03T11:44:30.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.944+0000 7f1c7f93a800 20 add_watcher() i=7 2023-12-03T11:44:30.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.944+0000 7f1c7f93a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.947+0000 7f1c417e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.947+0000 7f1c7f93a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.947+0000 7f1c7f93a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.953+0000 7f1c7f93a800 5 note: GC not initialized 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.953+0000 7f1baffff640 20 reqs_thread_entry: start 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.953+0000 7f1c7f93a800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.954+0000 7f1c7f93a800 20 Filter name: none 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.954+0000 7f1badffb640 20 reqs_thread_entry: start 2023-12-03T11:44:30.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.954+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.954+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:30.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.954+0000 7f1c7f93a800 20 rados->read ofs=0 len=0 2023-12-03T11:44:30.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:30.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:30.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:30.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 INFO: main: starting shard=0 2023-12-03T11:44:30.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:30.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 INFO: main: starting shard=1 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.955+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 INFO: main: starting shard=2 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 INFO: main: starting shard=3 2023-12-03T11:44:30.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.956+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 INFO: main: starting shard=4 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 INFO: main: starting shard=5 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.957+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:30.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.958+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.958+0000 7f1c7f93a800 20 INFO: main: starting shard=6 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.958+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 INFO: main: starting shard=7 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:30.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 INFO: main: starting shard=8 2023-12-03T11:44:30.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.959+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:30.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:44:30.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 INFO: main: starting shard=9 2023-12-03T11:44:30.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:30.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 INFO: main: starting shard=10 2023-12-03T11:44:30.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.960+0000 7f1c7f93a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:30.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.961+0000 7f1c7f93a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:30.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.961+0000 7f1c7f93a800 20 INFO: main: done 2023-12-03T11:44:30.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.962+0000 7f1c417e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:30.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.964+0000 7f1c7f93a800 20 remove_watcher() i=0 2023-12-03T11:44:30.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.964+0000 7f1c7f93a800 2 removed watcher, disabling cache 2023-12-03T11:44:30.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.965+0000 7f1c7f93a800 20 remove_watcher() i=1 2023-12-03T11:44:30.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.966+0000 7f1c7f93a800 20 remove_watcher() i=2 2023-12-03T11:44:30.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.967+0000 7f1c7f93a800 20 remove_watcher() i=3 2023-12-03T11:44:30.969 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.968+0000 7f1c7f93a800 20 remove_watcher() i=4 2023-12-03T11:44:30.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.970+0000 7f1c7f93a800 20 remove_watcher() i=5 2023-12-03T11:44:30.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.973+0000 7f1c7f93a800 20 remove_watcher() i=6 2023-12-03T11:44:30.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:30.974+0000 7f1c7f93a800 20 remove_watcher() i=7 2023-12-03T11:44:30.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:30.982 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:30.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:30.985 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:30.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:30.988 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.050 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:31.051 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:44:31.080 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:31.080 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.093+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 realm 2023-12-03T11:44:31.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.095+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.096+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.096+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.096+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:31.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.096+0000 7f7dad00f800 20 searching for the correct realm 2023-12-03T11:44:31.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.106+0000 7f7da85b1640 0 -- 172.21.15.116:0/4199278154 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557404bd8850 msgr2=0x557404bdac40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.107+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 zone default found 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 4 Realm: () 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 zonegroup default 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.108+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.109+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.109+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.109+0000 7f7dad00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.109+0000 7f7dad00f800 20 started sync module instance, tier type = 2023-12-03T11:44:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.109+0000 7f7dad00f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:31.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.120+0000 7f7da8db2640 0 -- 172.21.15.116:0/4199278154 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f7d9c06e400 msgr2=0x7f7d9c06e880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=0 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=1 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=2 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=3 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=4 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=5 2023-12-03T11:44:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.128+0000 7f7dad00f800 20 add_watcher() i=6 2023-12-03T11:44:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.132+0000 7f7dad00f800 20 add_watcher() i=7 2023-12-03T11:44:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.132+0000 7f7dad00f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.134+0000 7f7d6e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.134+0000 7f7dad00f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.134+0000 7f7dad00f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7dad00f800 5 note: GC not initialized 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7cdd7fa640 20 reqs_thread_entry: start 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7dad00f800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7dad00f800 20 Filter name: none 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7cd6ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:31.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.140+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:31.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 INFO: main: starting shard=0 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.141+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: starting shard=1 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: starting shard=2 2023-12-03T11:44:31.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 INFO: main: starting shard=3 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.142+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 INFO: main: starting shard=4 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 INFO: main: starting shard=5 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.143+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:44:31.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 INFO: main: starting shard=6 2023-12-03T11:44:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 INFO: main: starting shard=7 2023-12-03T11:44:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.144+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 INFO: main: starting shard=8 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 INFO: main: starting shard=9 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.145+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.146+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.146+0000 7f7dad00f800 20 INFO: main: starting shard=10 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.146+0000 7f7dad00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.146+0000 7f7dad00f800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.146+0000 7f7dad00f800 20 INFO: main: done 2023-12-03T11:44:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.147+0000 7f7d6e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.149+0000 7f7dad00f800 20 remove_watcher() i=0 2023-12-03T11:44:31.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.149+0000 7f7dad00f800 2 removed watcher, disabling cache 2023-12-03T11:44:31.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.150+0000 7f7dad00f800 20 remove_watcher() i=1 2023-12-03T11:44:31.151 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.151+0000 7f7dad00f800 20 remove_watcher() i=2 2023-12-03T11:44:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.152+0000 7f7dad00f800 20 remove_watcher() i=3 2023-12-03T11:44:31.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.153+0000 7f7dad00f800 20 remove_watcher() i=4 2023-12-03T11:44:31.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.155+0000 7f7dad00f800 20 remove_watcher() i=5 2023-12-03T11:44:31.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.156+0000 7f7dad00f800 20 remove_watcher() i=6 2023-12-03T11:44:31.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.157+0000 7f7dad00f800 20 remove_watcher() i=7 2023-12-03T11:44:31.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:31.165 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:31.168 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.224 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:31.226 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-12-03T11:44:31.254 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:31.255 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:31.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.266+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 realm 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.268+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.269+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.269+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.269+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:31.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.269+0000 7f9cc1325800 20 searching for the correct realm 2023-12-03T11:44:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.279+0000 7f9cbcf4d640 0 -- 172.21.15.116:0/2917219648 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x560d3db48680 msgr2=0x560d3db4aa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.280+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:31.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 zone default found 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 4 Realm: () 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 zonegroup default 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.281+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.282+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.282+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.282+0000 7f9cc1325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.282+0000 7f9cc1325800 20 started sync module instance, tier type = 2023-12-03T11:44:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.282+0000 7f9cc1325800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:31.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.288+0000 7f9cbd74e640 0 -- 172.21.15.116:0/2917219648 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9ca40568f0 msgr2=0x7f9ca4058730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=0 2023-12-03T11:44:31.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=1 2023-12-03T11:44:31.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=2 2023-12-03T11:44:31.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=3 2023-12-03T11:44:31.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=4 2023-12-03T11:44:31.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=5 2023-12-03T11:44:31.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.295+0000 7f9cc1325800 20 add_watcher() i=6 2023-12-03T11:44:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.297+0000 7f9cc1325800 20 add_watcher() i=7 2023-12-03T11:44:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.297+0000 7f9cc1325800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:31.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.299+0000 7f9c837de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.300+0000 7f9cc1325800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:31.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.300+0000 7f9cc1325800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.305+0000 7f9cc1325800 5 note: GC not initialized 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.305+0000 7f9bf27fc640 20 reqs_thread_entry: start 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.305+0000 7f9cc1325800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.306+0000 7f9cc1325800 20 Filter name: none 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.306+0000 7f9be3fff640 20 reqs_thread_entry: start 2023-12-03T11:44:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.306+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.306+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.306+0000 7f9cc1325800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 INFO: main: starting shard=0 2023-12-03T11:44:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 INFO: main: starting shard=1 2023-12-03T11:44:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.307+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:44:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: starting shard=2 2023-12-03T11:44:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: starting shard=3 2023-12-03T11:44:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 INFO: main: starting shard=4 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.308+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 INFO: main: starting shard=5 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 INFO: main: starting shard=6 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.309+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: starting shard=7 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: starting shard=8 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 INFO: main: starting shard=9 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.310+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.311+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.311+0000 7f9cc1325800 20 INFO: main: starting shard=10 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.311+0000 7f9cc1325800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.311+0000 7f9cc1325800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.311+0000 7f9cc1325800 20 INFO: main: done 2023-12-03T11:44:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.312+0000 7f9c837de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.317+0000 7f9cc1325800 20 remove_watcher() i=0 2023-12-03T11:44:31.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.317+0000 7f9cc1325800 2 removed watcher, disabling cache 2023-12-03T11:44:31.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.319+0000 7f9cc1325800 20 remove_watcher() i=1 2023-12-03T11:44:31.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.320+0000 7f9cc1325800 20 remove_watcher() i=2 2023-12-03T11:44:31.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.321+0000 7f9cc1325800 20 remove_watcher() i=3 2023-12-03T11:44:31.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.322+0000 7f9cc1325800 20 remove_watcher() i=4 2023-12-03T11:44:31.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.323+0000 7f9cc1325800 20 remove_watcher() i=5 2023-12-03T11:44:31.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.325+0000 7f9cc1325800 20 remove_watcher() i=6 2023-12-03T11:44:31.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.326+0000 7f9cc1325800 20 remove_watcher() i=7 2023-12-03T11:44:31.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:31.335 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:31.337 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.389 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:44:31.390 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-12-03T11:44:31.420 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:31.420 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:31.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.432+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 realm 2023-12-03T11:44:31.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.434+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.435+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.435+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.435+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:31.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.435+0000 7f9507212800 20 searching for the correct realm 2023-12-03T11:44:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.646+0000 7f9502857640 0 -- 172.21.15.116:0/845045937 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x563f65bbcf80 msgr2=0x563f65bbf370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.647+0000 7f9507212800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.647+0000 7f9507212800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.647+0000 7f9507212800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.647+0000 7f9507212800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.647+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:31.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 zone default found 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 4 Realm: () 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 zonegroup default 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.648+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.649+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.649+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.649+0000 7f9507212800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.649+0000 7f9507212800 20 started sync module instance, tier type = 2023-12-03T11:44:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.649+0000 7f9507212800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:31.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.657+0000 7f9502056640 0 -- 172.21.15.116:0/845045937 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x563f65bc1f60 msgr2=0x563f65bc4350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=0 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=1 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=2 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=3 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=4 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.660+0000 7f9507212800 20 add_watcher() i=5 2023-12-03T11:44:31.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.661+0000 7f9507212800 20 add_watcher() i=6 2023-12-03T11:44:31.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.661+0000 7f9507212800 20 add_watcher() i=7 2023-12-03T11:44:31.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.661+0000 7f9507212800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.663+0000 7f9443fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.663+0000 7f9507212800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.663+0000 7f9507212800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9507212800 5 note: GC not initialized 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9432ffd640 20 reqs_thread_entry: start 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9507212800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9507212800 20 Filter name: none 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9430ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:31.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.669+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 rados->read ofs=0 len=0 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=11 2023-12-03T11:44:31.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 INFO: main: starting shard=0 2023-12-03T11:44:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.670+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 INFO: main: starting shard=1 2023-12-03T11:44:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 INFO: main: starting shard=2 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.671+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: starting shard=3 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: starting shard=4 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 INFO: main: starting shard=5 2023-12-03T11:44:31.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.672+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:31.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 INFO: main: starting shard=6 2023-12-03T11:44:31.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:31.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 INFO: main: starting shard=7 2023-12-03T11:44:31.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.673+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:31.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 INFO: main: starting shard=8 2023-12-03T11:44:31.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 INFO: main: starting shard=9 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.674+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.675+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.675+0000 7f9507212800 20 INFO: main: starting shard=10 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.675+0000 7f9507212800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.675+0000 7f9507212800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:44:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.675+0000 7f9507212800 20 INFO: main: done 2023-12-03T11:44:31.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.676+0000 7f9443fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.678+0000 7f9507212800 20 remove_watcher() i=0 2023-12-03T11:44:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.678+0000 7f9507212800 2 removed watcher, disabling cache 2023-12-03T11:44:31.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.679+0000 7f9507212800 20 remove_watcher() i=1 2023-12-03T11:44:31.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.680+0000 7f9507212800 20 remove_watcher() i=2 2023-12-03T11:44:31.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.681+0000 7f9507212800 20 remove_watcher() i=3 2023-12-03T11:44:31.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.682+0000 7f9507212800 20 remove_watcher() i=4 2023-12-03T11:44:31.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.683+0000 7f9507212800 20 remove_watcher() i=5 2023-12-03T11:44:31.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.684+0000 7f9507212800 20 remove_watcher() i=6 2023-12-03T11:44:31.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:31.685+0000 7f9507212800 20 remove_watcher() i=7 2023-12-03T11:44:31.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:31.693 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:31.696 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:44:32.073 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:44:32.448 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2023-12-03T11:44:32.476 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:32.476 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:32.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.488+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.489+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.489+0000 7faabc88c800 20 realm 2023-12-03T11:44:32.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.489+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.490+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.490+0000 7faabc88c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:32.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.490+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.490+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.490+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.491+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:32.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.491+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.491+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:32.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.491+0000 7faabc88c800 20 searching for the correct realm 2023-12-03T11:44:32.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faaba38c640 0 -- 172.21.15.116:0/205831035 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557b258e26d0 msgr2=0x557b258e4b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.503+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:32.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 zone default found 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 4 Realm: () 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 zonegroup default 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.504+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.505+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.505+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.505+0000 7faabc88c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.505+0000 7faabc88c800 20 started sync module instance, tier type = 2023-12-03T11:44:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.505+0000 7faabc88c800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:32.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.516+0000 7faab3fff640 0 -- 172.21.15.116:0/205831035 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x557b258e76f0 msgr2=0x557b258e9ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=0 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=1 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=2 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=3 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=4 2023-12-03T11:44:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.517+0000 7faabc88c800 20 add_watcher() i=5 2023-12-03T11:44:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.518+0000 7faabc88c800 20 add_watcher() i=6 2023-12-03T11:44:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.518+0000 7faabc88c800 20 add_watcher() i=7 2023-12-03T11:44:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.518+0000 7faabc88c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.521+0000 7faa7ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.521+0000 7faabc88c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.521+0000 7faabc88c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:32.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 5 note: GC not initialized 2023-12-03T11:44:32.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7fa9eeffd640 20 reqs_thread_entry: start 2023-12-03T11:44:32.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:32.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 20 Filter name: none 2023-12-03T11:44:32.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7fa9ecff9640 20 reqs_thread_entry: start 2023-12-03T11:44:32.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:32.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.527+0000 7faabc88c800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:32.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.528+0000 7faabc88c800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.529+0000 7faabc88c800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.529+0000 7faabc88c800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.529+0000 7faabc88c800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:44:32.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.529+0000 7faabc88c800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:44:32.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.535+0000 7faa7ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:32.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.537+0000 7faabc88c800 20 remove_watcher() i=0 2023-12-03T11:44:32.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.537+0000 7faabc88c800 2 removed watcher, disabling cache 2023-12-03T11:44:32.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.538+0000 7faabc88c800 20 remove_watcher() i=1 2023-12-03T11:44:32.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.539+0000 7faabc88c800 20 remove_watcher() i=2 2023-12-03T11:44:32.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.541+0000 7faabc88c800 20 remove_watcher() i=3 2023-12-03T11:44:32.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.542+0000 7faabc88c800 20 remove_watcher() i=4 2023-12-03T11:44:32.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.543+0000 7faabc88c800 20 remove_watcher() i=5 2023-12-03T11:44:32.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.544+0000 7faabc88c800 20 remove_watcher() i=6 2023-12-03T11:44:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.545+0000 7faabc88c800 20 remove_watcher() i=7 2023-12-03T11:44:32.552 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:44:32.585 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:32.585 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:32.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.598+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 20 realm 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.600+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.601+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.601+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.602+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.602+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.602+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.602+0000 7f259293b800 20 searching for the correct realm 2023-12-03T11:44:32.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.608+0000 7f258e640640 0 -- 172.21.15.116:0/1882678531 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f257805c420 msgr2=0x7f257805c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:32.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.814+0000 7f259293b800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:32.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.814+0000 7f259293b800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:32.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.814+0000 7f259293b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:32.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.814+0000 7f259293b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.814+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f2585e3f640 0 -- 172.21.15.116:0/1882678531 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x55af9f2ff780 msgr2=0x55af9f301ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 zone default found 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 4 Realm: () 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 zonegroup default 2023-12-03T11:44:32.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.815+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 started sync module instance, tier type = 2023-12-03T11:44:32.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.816+0000 7f259293b800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=0 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=1 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=2 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=3 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=4 2023-12-03T11:44:32.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.828+0000 7f259293b800 20 add_watcher() i=5 2023-12-03T11:44:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.831+0000 7f259293b800 20 add_watcher() i=6 2023-12-03T11:44:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.831+0000 7f259293b800 20 add_watcher() i=7 2023-12-03T11:44:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.831+0000 7f259293b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:32.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.834+0000 7f2553fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:32.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.834+0000 7f259293b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:32.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.834+0000 7f259293b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f259293b800 5 note: GC not initialized 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f254affd640 20 reqs_thread_entry: start 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f259293b800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f259293b800 20 Filter name: none 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f2548ff9640 20 reqs_thread_entry: start 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f259293b800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.840+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:32.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:44:32.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:44:32.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 : miss 2023-12-03T11:44:32.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 20 rados->read ofs=0 len=0 2023-12-03T11:44:32.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 info.flags=0x17 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 to cache LRU end 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.841+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.0 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.842+0000 7f259293b800 1 NOTICE: finished shard 0 (0 entries removed) 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.842+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.843+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.2 2023-12-03T11:44:32.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.843+0000 7f259293b800 1 NOTICE: finished shard 2 (0 entries removed) 2023-12-03T11:44:32.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.843+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.3 2023-12-03T11:44:32.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.844+0000 7f259293b800 1 NOTICE: finished shard 3 (0 entries removed) 2023-12-03T11:44:32.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.844+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.4 2023-12-03T11:44:32.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.844+0000 7f259293b800 1 NOTICE: finished shard 4 (0 entries removed) 2023-12-03T11:44:32.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.844+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.845+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.6 2023-12-03T11:44:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.845+0000 7f259293b800 1 NOTICE: finished shard 6 (0 entries removed) 2023-12-03T11:44:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.845+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.7 2023-12-03T11:44:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.846+0000 7f259293b800 1 NOTICE: finished shard 7 (0 entries removed) 2023-12-03T11:44:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.846+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.846+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.9 2023-12-03T11:44:32.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.847+0000 7f259293b800 1 NOTICE: finished shard 9 (0 entries removed) 2023-12-03T11:44:32.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.847+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.847+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.847+0000 7f259293b800 20 get_obj_state: octx=0x7f2580024b10 obj=check-bucket:e state=0x55af9f3dcf28 s->prefetch_data=0 2023-12-03T11:44:32.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.847+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.848+0000 7f259293b800 20 get_obj_state: octx=0x7f2588095040 obj=check-bucket:d state=0x55af9f41f148 s->prefetch_data=0 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.848+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.848+0000 7f259293b800 20 get_obj_state: octx=0x7f258003f830 obj=check-bucket:c state=0x55af9f41fc98 s->prefetch_data=0 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.848+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state: octx=0x55af9f0ab0f0 obj=check-bucket:f state=0x55af9f4207e8 s->prefetch_data=0 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to ya84b45wio348uxo5n1wf05fbcrmfqjy 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state_impl: setting s->olh_tag to sqwszkkxxrw0b2g1yhti4j5ri6225vu2 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=sqwszkkxxrw0b2g1yhti4j5ri6225vu2 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to fkw7vzo6j3fsso64qbphp989fe1tcaoc 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state_impl: setting s->olh_tag to el95rtttcnv5za37yob17vyla7sl4x87 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=el95rtttcnv5za37yob17vyla7sl4x87 2023-12-03T11:44:32.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to g0dyicq2tivf2ev7yne3mh9kgr723amq 2023-12-03T11:44:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state_impl: setting s->olh_tag to avv0qf1xiy84phxjz4c7ahp8annld7f5 2023-12-03T11:44:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=avv0qf1xiy84phxjz4c7ahp8annld7f5 2023-12-03T11:44:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to 2q5zm3h2m3odtle8d5xub1a5i666w9kb 2023-12-03T11:44:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 get_obj_state_impl: setting s->olh_tag to eqngs6y4edse8mktzwj46h4hdcq4me5z 2023-12-03T11:44:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.849+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=eqngs6y4edse8mktzwj46h4hdcq4me5z 2023-12-03T11:44:32.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.850+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.851+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.853+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.853+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 olh_log_entry: epoch=2 op=3 key=e[a8-XB.ADXqJ5qflTrApPUkZHjAkGiRE] 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: octx=0x7f2580024b10 obj=check-bucket:_:a8-XB.ADXqJ5qflTrApPUkZHjAkGiRE_e state=0x55af9f424558 s->prefetch_data=0 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 olh_log_entry: epoch=3 op=3 key=d[2lEamhloGHtEZTW4y1dt.sLBfj6Jl5B] 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: octx=0x7f2588095040 obj=check-bucket:_:2lEamhloGHtEZTW4y1dt.sLBfj6Jl5B_d state=0x55af9f43edc8 s->prefetch_data=0 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 10 manifest: total_size = 11 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1737870050317819020 2023-12-03T11:44:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: octx=0x7f2588095040 obj=check-bucket:_:2lEamhloGHtEZTW4y1dt.sLBfj6Jl5B_d state=0x55af9f43edc8 s->prefetch_data=0 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 prepare_atomic_modification: state is not atomic. state=0x55af9f43edc8 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 10 manifest: total_size = 11 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4164373012842536381 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 get_obj_state: octx=0x7f2580024b10 obj=check-bucket:_:a8-XB.ADXqJ5qflTrApPUkZHjAkGiRE_e state=0x55af9f424558 s->prefetch_data=0 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 prepare_atomic_modification: state is not atomic. state=0x55af9f424558 2023-12-03T11:44:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.856+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.858+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.858+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-12-03T11:44:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 olh_log_entry: epoch=2 op=3 key=f[FfZvDmYXWwCGytvYKV39HAcroDKMiGA] 2023-12-03T11:44:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: octx=0x55af9f0ab0f0 obj=check-bucket:_:FfZvDmYXWwCGytvYKV39HAcroDKMiGA_f state=0x55af9f44a4d8 s->prefetch_data=0 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 olh_log_entry: epoch=3 op=3 key=c[e7.KXCzPv5oVMc4dPqO4HcwhfpGzIv1] 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: octx=0x7f258003f830 obj=check-bucket:_:e7.KXCzPv5oVMc4dPqO4HcwhfpGzIv1_c state=0x55af9f44f068 s->prefetch_data=0 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 10 manifest: total_size = 12 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2861188006162583960 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: octx=0x55af9f0ab0f0 obj=check-bucket:_:FfZvDmYXWwCGytvYKV39HAcroDKMiGA_f state=0x55af9f44a4d8 s->prefetch_data=0 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 prepare_atomic_modification: state is not atomic. state=0x55af9f44a4d8 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 10 manifest: total_size = 22 2023-12-03T11:44:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.12314424673264892210 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 get_obj_state: octx=0x7f258003f830 obj=check-bucket:_:e7.KXCzPv5oVMc4dPqO4HcwhfpGzIv1_c state=0x55af9f44f068 s->prefetch_data=0 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 prepare_atomic_modification: state is not atomic. state=0x55af9f44f068 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.859+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.861+0000 7f259293b800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.861+0000 7f259293b800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:44:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.861+0000 7f259293b800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:32.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.861+0000 7f259293b800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:a8-XB.ADXqJ5qflTrApPUkZHjAkGiRE_.AO_O6RvlIrF1QgKx8_BlTm4AlD96554_0 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.864+0000 7f256ffff640 20 handle_completion(): completion ok for obj=e[a8-XB.ADXqJ5qflTrApPUkZHjAkGiRE] 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.864+0000 7f259293b800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.865+0000 7f259293b800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.865+0000 7f259293b800 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.865+0000 7f259293b800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.865+0000 7f259293b800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:FfZvDmYXWwCGytvYKV39HAcroDKMiGA_.o2-vTpo5EAOLJWc0gN-NutMmt2SfsDp_0 2023-12-03T11:44:32.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.866+0000 7f256ffff640 20 handle_completion(): completion ok for obj=f[FfZvDmYXWwCGytvYKV39HAcroDKMiGA] 2023-12-03T11:44:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.869+0000 7f259293b800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.869+0000 7f259293b800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.869+0000 7f259293b800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:44:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.869+0000 7f259293b800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.869+0000 7f259293b800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:e7.KXCzPv5oVMc4dPqO4HcwhfpGzIv1_.fGcG0H1okKaEXHWptLAhyfSigwTjUxv_0 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f2584e3d640 20 handle_completion(): completion ok for obj=c[e7.KXCzPv5oVMc4dPqO4HcwhfpGzIv1] 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f259293b800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f259293b800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f259293b800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f259293b800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:44:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.874+0000 7f259293b800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.1__shadow:2lEamhloGHtEZTW4y1dt.sLBfj6Jl5B_.sjtjFqoy__GRO5L91gldZPJVnTekfKR_0 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f259293b800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f259293b800 20 get_obj_state: octx=0x7f2580024b10 obj=check-bucket:e state=0x55af9f3dcf28 s->prefetch_data=0 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=sqwszkkxxrw0b2g1yhti4j5ri6225vu2 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f259293b800 20 get_obj_state: octx=0x55af9f0ab0f0 obj=check-bucket:f state=0x55af9f4207e8 s->prefetch_data=0 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f259293b800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=eqngs6y4edse8mktzwj46h4hdcq4me5z 2023-12-03T11:44:32.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.876+0000 7f256ffff640 20 handle_completion(): completion ok for obj=d[2lEamhloGHtEZTW4y1dt.sLBfj6Jl5B] 2023-12-03T11:44:32.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.880+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.880+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.880+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.880+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.881+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.5 2023-12-03T11:44:32.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.881+0000 7f259293b800 1 NOTICE: finished shard 5 (1 entries removed) 2023-12-03T11:44:32.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.883+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.8 2023-12-03T11:44:32.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.883+0000 7f259293b800 1 NOTICE: finished shard 8 (1 entries removed) 2023-12-03T11:44:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.888+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.888+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.890+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.1 2023-12-03T11:44:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.890+0000 7f259293b800 1 NOTICE: finished shard 1 (1 entries removed) 2023-12-03T11:44:32.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.891+0000 7f259293b800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.1.10 2023-12-03T11:44:32.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.891+0000 7f259293b800 1 NOTICE: finished shard 10 (1 entries removed) 2023-12-03T11:44:32.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.891+0000 7f259293b800 1 NOTICE: finished all shards (4 entries removed) 2023-12-03T11:44:32.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.892+0000 7f2553fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:32.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.895+0000 7f259293b800 20 remove_watcher() i=0 2023-12-03T11:44:32.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.895+0000 7f259293b800 2 removed watcher, disabling cache 2023-12-03T11:44:32.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.896+0000 7f259293b800 20 remove_watcher() i=1 2023-12-03T11:44:32.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.897+0000 7f259293b800 20 remove_watcher() i=2 2023-12-03T11:44:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.898+0000 7f259293b800 20 remove_watcher() i=3 2023-12-03T11:44:32.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.899+0000 7f259293b800 20 remove_watcher() i=4 2023-12-03T11:44:32.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.900+0000 7f259293b800 20 remove_watcher() i=5 2023-12-03T11:44:32.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.901+0000 7f259293b800 20 remove_watcher() i=6 2023-12-03T11:44:32.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.902+0000 7f259293b800 20 remove_watcher() i=7 2023-12-03T11:44:32.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:32.909 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:32.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:32.915 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket check-bucket 2023-12-03T11:44:32.985 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:44:32.985 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:44:32.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.997+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.999+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:32.999+0000 7f951e135800 20 realm 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.000+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.001+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:33.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.001+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.001+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:44:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.001+0000 7f951e135800 20 searching for the correct realm 2023-12-03T11:44:33.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.010+0000 7f95196cc640 0 -- 172.21.15.116:0/3583023804 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55ed7d46b930 msgr2=0x55ed7d46dd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.011+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:44:33.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 zone default found 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 4 Realm: () 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 zonegroup default 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.012+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.013+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.013+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.013+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.013+0000 7f951e135800 20 started sync module instance, tier type = 2023-12-03T11:44:33.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.013+0000 7f951e135800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:44:33.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.017+0000 7f9519ecd640 0 -- 172.21.15.116:0/3583023804 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f950c059340 msgr2=0x7f950c057740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:44:33.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=0 2023-12-03T11:44:33.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=1 2023-12-03T11:44:33.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=2 2023-12-03T11:44:33.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=3 2023-12-03T11:44:33.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=4 2023-12-03T11:44:33.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.027+0000 7f951e135800 20 add_watcher() i=5 2023-12-03T11:44:33.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.029+0000 7f951e135800 20 add_watcher() i=6 2023-12-03T11:44:33.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.029+0000 7f951e135800 20 add_watcher() i=7 2023-12-03T11:44:33.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.029+0000 7f951e135800 2 all 8 watchers are set, enabling cache 2023-12-03T11:44:33.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.032+0000 7f94df7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:33.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.032+0000 7f951e135800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:44:33.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.032+0000 7f951e135800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:44:33.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f951e135800 5 note: GC not initialized 2023-12-03T11:44:33.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f944e7fc640 20 reqs_thread_entry: start 2023-12-03T11:44:33.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f951e135800 20 init_complete bucket index max shards: 11 2023-12-03T11:44:33.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f951e135800 20 Filter name: none 2023-12-03T11:44:33.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f943ffff640 20 reqs_thread_entry: start 2023-12-03T11:44:33.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.038+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.039+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.040+0000 7f951e135800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:44:33.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.040+0000 7f951e135800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:44:33.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.040+0000 7f951e135800 20 rados->read ofs=0 len=0 2023-12-03T11:44:33.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.041+0000 7f951e135800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:44:33.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.041+0000 7f94df7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:44:33.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.043+0000 7f951e135800 20 remove_watcher() i=0 2023-12-03T11:44:33.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.043+0000 7f951e135800 2 removed watcher, disabling cache 2023-12-03T11:44:33.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.044+0000 7f951e135800 20 remove_watcher() i=1 2023-12-03T11:44:33.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.045+0000 7f951e135800 20 remove_watcher() i=2 2023-12-03T11:44:33.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.046+0000 7f951e135800 20 remove_watcher() i=3 2023-12-03T11:44:33.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.047+0000 7f951e135800 20 remove_watcher() i=4 2023-12-03T11:44:33.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.048+0000 7f951e135800 20 remove_watcher() i=5 2023-12-03T11:44:33.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.050+0000 7f951e135800 20 remove_watcher() i=6 2023-12-03T11:44:33.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:44:33.051+0000 7f951e135800 20 remove_watcher() i=7 2023-12-03T11:44:33.057 INFO:tasks.workunit.client.0.smithi116.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2023-12-03T11:44:33.057 INFO:tasks.workunit.client.0.smithi116.stderr:Deleting bucket check-bucket 2023-12-03T11:44:33.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:33.059 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:33.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:44:33.063 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:44:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:Completed bucket check tests 2023-12-03T11:44:33.093 INFO:tasks.workunit.client.0.smithi116.stdout:Could not connect to the endpoint URL: "http://localhost:80/" 2023-12-03T11:44:33.093 INFO:tasks.workunit.client.0.smithi116.stdout:Could not connect to the endpoint URL: "http://localhost:8000/" 2023-12-03T11:44:33.093 INFO:tasks.workunit.client.0.smithi116.stdout:connected to https://localhost:443 2023-12-03T11:44:33.151 INFO:tasks.workunit.client.0.smithi116.stderr:+ deactivate 2023-12-03T11:44:33.151 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin ']' 2023-12-03T11:44:33.151 INFO:tasks.workunit.client.0.smithi116.stderr:+ PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ export PATH 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset _OLD_VIRTUAL_PATH 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n '' ']' 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ hash -r 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n '' ']' 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset VIRTUAL_ENV 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' '!' '' = nondestructive ']' 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset -f deactivate 2023-12-03T11:44:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:+ echo OK. 2023-12-03T11:44:33.153 INFO:tasks.workunit.client.0.smithi116.stdout:OK. 2023-12-03T11:44:33.153 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:44:33.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:44:33.222 INFO:tasks.workunit:Stopping ['rgw/run-bucket-check.sh'] on client.0... 2023-12-03T11:44:33.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:44:33.546 DEBUG:teuthology.parallel:result is None 2023-12-03T11:44:33.547 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:44:33.574 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:44:33.574 DEBUG:teuthology.orchestra.run.smithi116:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:44:33.636 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:44:33.637 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:44:33.652 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:44:33.652 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:44:33.652 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:44:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:44:33.692 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:44:33.692 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:44:33.747 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:44:33.747 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:44:33.802 INFO:tasks.workunit:timeout=3h 2023-12-03T11:44:33.802 INFO:tasks.workunit:cleanup=True 2023-12-03T11:44:33.802 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:44:33.857 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:45:38.614 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:45:38.615 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:45:38.670 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:45:38.672 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:45:38.672 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:45:38.721 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:45:38.769 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:45:38.810 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:45:38.811 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:45:38.811 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:45:38.851 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:45:38.854 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:45:38.854 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:45:38.909 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2023-12-03T11:45:38.910 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2023-12-03T11:45:38.910 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2023-12-03T11:45:38.977 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:45:38.978 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:45:38.978 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:45:38.978 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from ClsLock 2023-12-03T11:45:38.978 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestMultiLocking 2023-12-03T11:45:40.838 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:40.838 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: exclusive 2023-12-03T11:45:40.838 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:40.838 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:40.838 INFO:tasks.workunit.client.0.smithi116.stdout: client.5099 expiration=2023-12-03T11:47:40.829765+0000 addr=v1:172.21.15.116:0/3196558442 cookie= 2023-12-03T11:45:40.845 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:40.845 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: shared 2023-12-03T11:45:40.846 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:40.846 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:40.846 INFO:tasks.workunit.client.0.smithi116.stdout: client.5099 expiration=2023-12-03T11:47:40.842410+0000 addr=v1:172.21.15.116:0/3196558442 cookie= 2023-12-03T11:45:40.846 INFO:tasks.workunit.client.0.smithi116.stdout: client.5105 expiration=0.000000 addr=v1:172.21.15.116:0/955801123 cookie= 2023-12-03T11:45:40.847 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:40.847 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: shared 2023-12-03T11:45:40.847 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:40.847 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:40.847 INFO:tasks.workunit.client.0.smithi116.stdout: client.5105 expiration=0.000000 addr=v1:172.21.15.116:0/955801123 cookie= 2023-12-03T11:45:41.809 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestMultiLocking (2831 ms) 2023-12-03T11:45:41.809 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestMeta 2023-12-03T11:45:43.854 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:43.854 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: shared 2023-12-03T11:45:43.854 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:43.854 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:43.854 INFO:tasks.workunit.client.0.smithi116.stdout: client.5114 expiration=0.000000 addr=v1:172.21.15.116:0/29112824 cookie= 2023-12-03T11:45:43.856 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:43.856 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: exclusive 2023-12-03T11:45:43.856 INFO:tasks.workunit.client.0.smithi116.stdout: tag: new_tag 2023-12-03T11:45:43.857 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:43.857 INFO:tasks.workunit.client.0.smithi116.stdout: client.5295 expiration=0.000000 addr=v1:172.21.15.116:0/1664308319 cookie= 2023-12-03T11:45:44.841 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestMeta (3032 ms) 2023-12-03T11:45:44.841 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestCookie 2023-12-03T11:45:46.873 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:46.873 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: exclusive 2023-12-03T11:45:46.873 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:46.873 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:46.878 INFO:tasks.workunit.client.0.smithi116.stdout:lock: mylock 2023-12-03T11:45:46.878 INFO:tasks.workunit.client.0.smithi116.stdout: lock_type: shared 2023-12-03T11:45:46.878 INFO:tasks.workunit.client.0.smithi116.stdout: tag: 2023-12-03T11:45:46.878 INFO:tasks.workunit.client.0.smithi116.stdout: lockers: 2023-12-03T11:45:46.879 INFO:tasks.workunit.client.0.smithi116.stdout: client.5120 expiration=0.000000 addr=v1:172.21.15.116:0/4168261013 cookie= 2023-12-03T11:45:46.879 INFO:tasks.workunit.client.0.smithi116.stdout: client.5120 expiration=0.000000 addr=v1:172.21.15.116:0/4168261013 cookie=new cookie 2023-12-03T11:45:47.860 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestCookie (3018 ms) 2023-12-03T11:45:47.860 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestMultipleLocks 2023-12-03T11:45:50.883 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestMultipleLocks (3023 ms) 2023-12-03T11:45:50.883 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestLockDuration 2023-12-03T11:45:57.963 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestLockDuration (7079 ms) 2023-12-03T11:45:57.963 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestAssertLocked 2023-12-03T11:46:00.980 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestAssertLocked (3018 ms) 2023-12-03T11:46:01.006 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestSetCookie 2023-12-03T11:46:04.003 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestSetCookie (3022 ms) 2023-12-03T11:46:04.004 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestRenew 2023-12-03T11:46:25.057 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestRenew (21055 ms) 2023-12-03T11:46:25.057 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2023-12-03T11:46:32.083 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7025 ms) 2023-12-03T11:46:32.083 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2023-12-03T11:46:39.789 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (7706 ms) 2023-12-03T11:46:39.789 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8035 ms) 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from ClsLock (68845 ms total) 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 11 tests from 1 test suite ran. (68845 ms total) 2023-12-03T11:46:47.823 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 11 tests. 2023-12-03T11:46:47.826 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:46:47.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:46:47.896 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2023-12-03T11:46:47.897 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2023-12-03T11:46:47.897 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2023-12-03T11:46:47.968 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:46:47.968 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 4 tests from 1 test suite. 2023-12-03T11:46:47.969 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:46:47.969 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from cls_log 2023-12-03T11:46:47.969 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_log.test_log_add_same_time 2023-12-03T11:46:50.854 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_log.test_log_add_same_time (2886 ms) 2023-12-03T11:46:50.854 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_log.test_log_add_different_time 2023-12-03T11:46:53.864 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_log.test_log_add_different_time (3010 ms) 2023-12-03T11:46:53.864 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_log.trim_by_time 2023-12-03T11:46:56.878 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_log.trim_by_time (3015 ms) 2023-12-03T11:46:56.879 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_log.trim_by_marker 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_log.trim_by_marker (3020 ms) 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from cls_log (11931 ms total) 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 4 tests from 1 test suite ran. (11931 ms total) 2023-12-03T11:46:59.899 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 4 tests. 2023-12-03T11:46:59.902 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:46:59.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:46:59.975 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2023-12-03T11:46:59.977 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2023-12-03T11:46:59.977 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2023-12-03T11:47:00.050 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:47:00.050 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 10 tests from 1 test suite. 2023-12-03T11:47:00.050 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:47:00.050 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 10 tests from cls_refcount 2023-12-03T11:47:00.050 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_implicit 2023-12-03T11:47:02.920 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_implicit (2869 ms) 2023-12-03T11:47:02.920 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_implicit_idempotent 2023-12-03T11:47:05.944 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_implicit_idempotent (3025 ms) 2023-12-03T11:47:05.945 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_put_snap 2023-12-03T11:47:14.994 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_put_snap (9049 ms) 2023-12-03T11:47:14.994 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_explicit 2023-12-03T11:47:18.002 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_explicit (3009 ms) 2023-12-03T11:47:18.003 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.set 2023-12-03T11:47:21.039 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.set (3036 ms) 2023-12-03T11:47:21.039 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_implicit_ec 2023-12-03T11:47:28.049 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_implicit_ec (7010 ms) 2023-12-03T11:47:28.049 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_implicit_idempotent_ec 2023-12-03T11:47:35.096 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_implicit_idempotent_ec (7048 ms) 2023-12-03T11:47:35.096 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_put_snap_ec 2023-12-03T11:47:47.813 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_put_snap_ec (12716 ms) 2023-12-03T11:47:47.813 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.test_explicit_ec 2023-12-03T11:47:54.839 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.test_explicit_ec (7026 ms) 2023-12-03T11:47:54.839 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_refcount.set_ec 2023-12-03T11:48:01.885 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_refcount.set_ec (7046 ms) 2023-12-03T11:48:01.885 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 10 tests from cls_refcount (61835 ms total) 2023-12-03T11:48:01.885 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:48:01.885 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:48:01.885 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 10 tests from 1 test suite ran. (61835 ms total) 2023-12-03T11:48:01.886 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 10 tests. 2023-12-03T11:48:01.888 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:48:01.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:48:01.959 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2023-12-03T11:48:01.960 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2023-12-03T11:48:01.960 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2023-12-03T11:48:02.031 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:48:02.032 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 16 tests from 1 test suite. 2023-12-03T11:48:02.032 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:48:02.032 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 16 tests from cls_rgw 2023-12-03T11:48:03.903 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_basic 2023-12-03T11:48:03.939 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_basic (36 ms) 2023-12-03T11:48:03.940 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2023-12-03T11:48:03.972 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (33 ms) 2023-12-03T11:48:03.972 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_remove_object 2023-12-03T11:48:04.020 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_remove_object (48 ms) 2023-12-03T11:48:04.021 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_suggest 2023-12-03T11:48:05.486 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_suggest (1465 ms) 2023-12-03T11:48:05.486 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_suggest_complete 2023-12-03T11:48:05.491 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_suggest_complete (6 ms) 2023-12-03T11:48:05.491 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_list 2023-12-03T11:48:05.512 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_list (20 ms) 2023-12-03T11:48:05.512 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_list_delimited 2023-12-03T11:48:25.003 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_list_delimited (19491 ms) 2023-12-03T11:48:25.003 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.bi_list 2023-12-03T11:48:25.074 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.bi_list (71 ms) 2023-12-03T11:48:25.074 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.gc_set 2023-12-03T11:48:25.083 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.gc_set (10 ms) 2023-12-03T11:48:25.083 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.gc_list 2023-12-03T11:48:25.093 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.gc_list (10 ms) 2023-12-03T11:48:25.093 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.gc_defer 2023-12-03T11:48:32.976 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.gc_defer (7883 ms) 2023-12-03T11:48:32.976 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.usage_basic 2023-12-03T11:48:33.059 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.usage_basic (83 ms) 2023-12-03T11:48:33.060 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2023-12-03T11:48:33.060 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.usage_clear_no_obj (1 ms) 2023-12-03T11:48:33.060 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.usage_clear 2023-12-03T11:48:33.159 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.usage_clear (99 ms) 2023-12-03T11:48:33.160 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.bi_log_trim 2023-12-03T11:48:33.197 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.bi_log_trim (37 ms) 2023-12-03T11:48:33.197 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw.index_racing_removes 2023-12-03T11:48:33.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw.index_racing_removes (13 ms) 2023-12-03T11:48:33.987 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 16 tests from cls_rgw (29306 ms total) 2023-12-03T11:48:33.988 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:48:33.988 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:48:33.988 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 16 tests from 1 test suite ran. (31954 ms total) 2023-12-03T11:48:33.988 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 16 tests. 2023-12-03T11:48:33.989 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:48:33.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:48:34.061 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_gc.sh on client.0... 2023-12-03T11:48:34.062 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_gc.sh... 2023-12-03T11:48:34.063 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_rgw_gc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_gc.sh 2023-12-03T11:48:34.129 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:48:34.130 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:48:34.130 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:48:34.130 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from cls_rgw_gc 2023-12-03T11:48:34.130 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.init 2023-12-03T11:48:35.996 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.init (1866 ms) 2023-12-03T11:48:35.996 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops1 2023-12-03T11:48:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:[ ] list info tag = chain-0 2023-12-03T11:48:36.023 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops1 (27 ms) 2023-12-03T11:48:36.023 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops2 2023-12-03T11:48:36.031 INFO:tasks.workunit.client.0.smithi116.stderr:[ ] next_marker is: = 0/4263 2023-12-03T11:48:36.032 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops2 (9 ms) 2023-12-03T11:48:36.032 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops5 2023-12-03T11:48:36.039 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops5 (8 ms) 2023-12-03T11:48:36.039 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops6 2023-12-03T11:48:36.046 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops6 (7 ms) 2023-12-03T11:48:36.046 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops7 2023-12-03T11:48:36.054 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops7 (7 ms) 2023-12-03T11:48:36.054 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops8 2023-12-03T11:48:36.062 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops8 (9 ms) 2023-12-03T11:48:36.062 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops9 2023-12-03T11:48:36.068 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.gc_queue_ops9 (6 ms) 2023-12-03T11:48:36.069 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_gc.finalize 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_gc.finalize (929 ms) 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from cls_rgw_gc (2868 ms total) 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 9 tests from 1 test suite ran. (2868 ms total) 2023-12-03T11:48:36.998 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 9 tests. 2023-12-03T11:48:37.000 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:48:37.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:48:37.074 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_stats.sh on client.0... 2023-12-03T11:48:37.075 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_stats.sh... 2023-12-03T11:48:37.075 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_rgw_stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_stats.sh 2023-12-03T11:48:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:48:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:48:37.153 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 1 test from 1 test suite. 2023-12-03T11:48:37.153 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:48:39.025 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 1 test from cls_rgw_stats 2023-12-03T11:48:39.026 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] cls_rgw_stats.simulate 2023-12-03T11:48:39.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.039+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=LJJ7FQ3RA501 type=del size=3248480 2023-12-03T11:48:39.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.041+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=YLXN9VDW770T type=del size=94516 2023-12-03T11:48:39.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.042+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=DGGPQG2HS2KN type=del size=1910199 2023-12-03T11:48:39.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.043+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=K715U152IKSA type=del size=3865915 2023-12-03T11:48:39.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.046+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~8FDAD3XYTXQ2.0 size=1048576 2023-12-03T11:48:39.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.047+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~8FDAD3XYTXQ2.1 size=586460 2023-12-03T11:48:39.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.047+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=8FDAD3XYTXQ2 tag=OXTUBIIYPCT0 type=write size=2683612 2023-12-03T11:48:39.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.048+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=D742WJAHXZWI type=write size=7008 2023-12-03T11:48:39.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.050+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=X78X2F3FUEZF type=del size=2410453 2023-12-03T11:48:39.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.051+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=Z20NPVVJ4JF5 type=del size=206851 2023-12-03T11:48:39.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.053+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=8FJ0U2JA7ACX type=del size=2930139 2023-12-03T11:48:39.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.055+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~YN1L0UVS60DZ.0 size=906378 2023-12-03T11:48:39.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.055+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=YN1L0UVS60DZ tag=GZ4WREMJPMNG type=write size=1954954 2023-12-03T11:48:39.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.057+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=AI7UK2OLJKUA type=write size=299452 2023-12-03T11:48:39.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.059+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=X85CY0RJM44T type=del size=3254795 2023-12-03T11:48:39.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.060+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=NS68DDKDTWFJ type=del size=708105 2023-12-03T11:48:39.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.061+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=9H3JH4LN9L2X type=del size=3950030 2023-12-03T11:48:39.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.063+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=4OY9PL49LJZQ type=del size=332859 2023-12-03T11:48:39.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.066+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~HJAXP6E3BS73.0 size=661049 2023-12-03T11:48:39.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.066+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=HJAXP6E3BS73 tag=L0WR3S6JJX2G type=write size=1709625 2023-12-03T11:48:39.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.068+0000 7f24574f5780 1 < suggested remove operation key=9LKFY1U0JRBQ tag=LJJ7FQ3RA501 type=del 2023-12-03T11:48:39.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.070+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=A3UP9IL943PC type=del size=3933388 2023-12-03T11:48:39.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.071+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=YLXN9VDW770T type=del 2023-12-03T11:48:39.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.072+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=BA9VS081X634 type=del size=3885509 2023-12-03T11:48:39.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.074+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=DGGPQG2HS2KN type=del 2023-12-03T11:48:39.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.076+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=K46GUVAQGPW5 type=del size=1192060 2023-12-03T11:48:39.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.077+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=K715U152IKSA type=del 2023-12-03T11:48:39.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.079+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~LWEQMHU29M7O.0 size=1048576 2023-12-03T11:48:39.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.080+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~LWEQMHU29M7O.1 size=111666 2023-12-03T11:48:39.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.080+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=LWEQMHU29M7O tag=N865Y1YGXRPA type=write size=2208818 2023-12-03T11:48:39.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.081+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~8FDAD3XYTXQ2.0 size=1048576 2023-12-03T11:48:39.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.082+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~8FDAD3XYTXQ2.1 size=586460 2023-12-03T11:48:39.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.084+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=8FDAD3XYTXQ2 tag=OXTUBIIYPCT0 type=write size=2683612 2023-12-03T11:48:39.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.085+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=O7PX56SM02ST type=del size=2186098 2023-12-03T11:48:39.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.087+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=D742WJAHXZWI type=write size=7008 2023-12-03T11:48:39.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.088+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=JE6HI9XEI712 type=del size=2002515 2023-12-03T11:48:39.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.089+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=X78X2F3FUEZF type=del 2023-12-03T11:48:39.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.090+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=N3SXXLHNHV0N type=del size=2073461 2023-12-03T11:48:39.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.092+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=Z20NPVVJ4JF5 type=del 2023-12-03T11:48:39.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.093+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=D8VQ9EEC1PSK type=del size=3056947 2023-12-03T11:48:39.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.094+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=8FJ0U2JA7ACX type=del 2023-12-03T11:48:39.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.095+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=J7UJL1EAL1IQ type=write size=735687 2023-12-03T11:48:39.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.097+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~YN1L0UVS60DZ.0 size=906378 2023-12-03T11:48:39.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.099+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=YN1L0UVS60DZ tag=GZ4WREMJPMNG type=write size=1954954 2023-12-03T11:48:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.100+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=79UIOEPK4VRY type=del size=667236 2023-12-03T11:48:39.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.101+0000 7f24574f5780 1 < completed write operation key=3EOEF9UMATIE tag=AI7UK2OLJKUA type=write size=299452 2023-12-03T11:48:39.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.102+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=SC7ML46GZWXC type=del size=2597689 2023-12-03T11:48:39.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.104+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=X85CY0RJM44T type=del 2023-12-03T11:48:39.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.106+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.0 size=1048576 2023-12-03T11:48:39.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.107+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.1 size=1048576 2023-12-03T11:48:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.108+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.2 size=741347 2023-12-03T11:48:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.108+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=RMSKNGQ6JBCC tag=ZZ64JRMSA9HD type=write size=3887075 2023-12-03T11:48:39.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.109+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=NS68DDKDTWFJ type=del 2023-12-03T11:48:39.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.111+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=KZ7OJ4PM8LFU type=del size=1733410 2023-12-03T11:48:39.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.112+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=9H3JH4LN9L2X type=del 2023-12-03T11:48:39.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.113+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=HOTV7L54BJ8X type=del size=1959868 2023-12-03T11:48:39.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.115+0000 7f24574f5780 1 < canceled operation key=WD7U31FW3NGZ tag=4OY9PL49LJZQ type=del size=332859 2023-12-03T11:48:39.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.116+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=E819G0MYVQF0 type=del size=487875 2023-12-03T11:48:39.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.117+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~HJAXP6E3BS73.0 size=661049 2023-12-03T11:48:39.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.119+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=HJAXP6E3BS73 tag=L0WR3S6JJX2G type=write size=1709625 2023-12-03T11:48:39.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.121+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~6NFXR2DAEBSZ.0 size=974684 2023-12-03T11:48:39.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.121+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=6NFXR2DAEBSZ tag=U3DY1EZDDEYQ type=write size=2023260 2023-12-03T11:48:39.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.123+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=A3UP9IL943PC type=del 2023-12-03T11:48:39.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.124+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=0M94R5B1VP16 type=del size=342027 2023-12-03T11:48:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.125+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=BA9VS081X634 type=del 2023-12-03T11:48:39.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.127+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=CTH8M5XB5WFC type=del size=3177253 2023-12-03T11:48:39.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.128+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=K46GUVAQGPW5 type=del 2023-12-03T11:48:39.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.130+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~5GFCTYGPFF4O.0 size=155128 2023-12-03T11:48:39.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.130+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=5GFCTYGPFF4O tag=W2BO6F3AFXKW type=write size=1203704 2023-12-03T11:48:39.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.131+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~LWEQMHU29M7O.0 size=1048576 2023-12-03T11:48:39.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.132+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~LWEQMHU29M7O.1 size=111666 2023-12-03T11:48:39.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.134+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=LWEQMHU29M7O tag=N865Y1YGXRPA type=write size=2208818 2023-12-03T11:48:39.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.135+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=4LKZZWSI8BXD type=del size=1498472 2023-12-03T11:48:39.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.136+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=O7PX56SM02ST type=del 2023-12-03T11:48:39.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.138+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=X9PPYPEHM2CZ type=del size=686455 2023-12-03T11:48:39.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.139+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=JE6HI9XEI712 type=del 2023-12-03T11:48:39.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.141+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~YK0QBHESXZQ2.0 size=1048576 2023-12-03T11:48:39.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.142+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~YK0QBHESXZQ2.1 size=745396 2023-12-03T11:48:39.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.142+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=YK0QBHESXZQ2 tag=W0IUA2YI5ZQF type=write size=2842548 2023-12-03T11:48:39.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.144+0000 7f24574f5780 1 < suggested update operation key=MY3K9M4QRS3L tag=N3SXXLHNHV0N type=del size=2073461 2023-12-03T11:48:39.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.145+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=QYDZ52BYM95D type=del size=3503132 2023-12-03T11:48:39.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.146+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=D8VQ9EEC1PSK type=del 2023-12-03T11:48:39.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.147+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=DO09V1YR0U6L type=del size=1451173 2023-12-03T11:48:39.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.149+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=J7UJL1EAL1IQ type=write size=735687 2023-12-03T11:48:39.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.151+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~JWR3EWYAMPRM.0 size=660978 2023-12-03T11:48:39.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.151+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=JWR3EWYAMPRM tag=8ZL1BL0B4U0X type=write size=1709554 2023-12-03T11:48:39.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.152+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=79UIOEPK4VRY type=del 2023-12-03T11:48:39.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.153+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=BCNJ131VTOI7 type=write size=520749 2023-12-03T11:48:39.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.154+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=SC7ML46GZWXC type=del 2023-12-03T11:48:39.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.156+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=WC18A08JZZ4J type=del size=38408 2023-12-03T11:48:39.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.157+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.0 size=1048576 2023-12-03T11:48:39.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.158+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.1 size=1048576 2023-12-03T11:48:39.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.159+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~RMSKNGQ6JBCC.2 size=741347 2023-12-03T11:48:39.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.161+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=RMSKNGQ6JBCC tag=ZZ64JRMSA9HD type=write size=3887075 2023-12-03T11:48:39.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.163+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.0 size=1048576 2023-12-03T11:48:39.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.164+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.1 size=1048576 2023-12-03T11:48:39.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.166+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.2 size=126390 2023-12-03T11:48:39.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.166+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=4SB5WJZ98O0I tag=UFRWFXEPVH88 type=write size=3272118 2023-12-03T11:48:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.167+0000 7f24574f5780 1 < canceled operation key=XRHVYVADRWET tag=KZ7OJ4PM8LFU type=del size=1733410 2023-12-03T11:48:39.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.169+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.0 size=1048576 2023-12-03T11:48:39.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.171+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.1 size=1048576 2023-12-03T11:48:39.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.172+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.2 size=11260 2023-12-03T11:48:39.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.172+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=6WZ6GKAZHXNX tag=M3Y9J6212P2N type=write size=3156988 2023-12-03T11:48:39.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.173+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=HOTV7L54BJ8X type=del 2023-12-03T11:48:39.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.174+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=YVGDDM5L7MMN type=del size=1315166 2023-12-03T11:48:39.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.175+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=E819G0MYVQF0 type=del 2023-12-03T11:48:39.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.177+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=2A89MW6OPKCK type=del size=3535253 2023-12-03T11:48:39.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.178+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~6NFXR2DAEBSZ.0 size=974684 2023-12-03T11:48:39.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.180+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=6NFXR2DAEBSZ tag=U3DY1EZDDEYQ type=write size=2023260 2023-12-03T11:48:39.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.181+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=JCGJ29VKK5VP type=del size=2280877 2023-12-03T11:48:39.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.182+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=0M94R5B1VP16 type=del 2023-12-03T11:48:39.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.184+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=CG7B26OF64TY type=del size=2200849 2023-12-03T11:48:39.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.185+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=CTH8M5XB5WFC type=del 2023-12-03T11:48:39.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.187+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~VT4NWG6U1A8I.0 size=394896 2023-12-03T11:48:39.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.187+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=VT4NWG6U1A8I tag=D1LGTL4N560O type=write size=1443472 2023-12-03T11:48:39.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.188+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~5GFCTYGPFF4O.0 size=155128 2023-12-03T11:48:39.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.191+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=5GFCTYGPFF4O tag=W2BO6F3AFXKW type=write size=1203704 2023-12-03T11:48:39.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.192+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=RAMD0PK8TPKD type=del size=73235 2023-12-03T11:48:39.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.193+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=4LKZZWSI8BXD type=del 2023-12-03T11:48:39.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.195+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~1FFFGM6FOFT8.0 size=1048576 2023-12-03T11:48:39.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.196+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~1FFFGM6FOFT8.1 size=617703 2023-12-03T11:48:39.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.196+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=1FFFGM6FOFT8 tag=3E8J7MX4OJGC type=write size=2714855 2023-12-03T11:48:39.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.198+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=X9PPYPEHM2CZ type=del 2023-12-03T11:48:39.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.199+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=G21FNBBGOX2I type=del size=1581394 2023-12-03T11:48:39.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.200+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~YK0QBHESXZQ2.0 size=1048576 2023-12-03T11:48:39.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.201+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~YK0QBHESXZQ2.1 size=745396 2023-12-03T11:48:39.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.203+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=YK0QBHESXZQ2 tag=W0IUA2YI5ZQF type=write size=2842548 2023-12-03T11:48:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.205+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~19G0DIMEF993.0 size=1048576 2023-12-03T11:48:39.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.206+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~19G0DIMEF993.1 size=248407 2023-12-03T11:48:39.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.206+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=19G0DIMEF993 tag=VBXJWLGWQ6D0 type=write size=2345559 2023-12-03T11:48:39.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.207+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=QYDZ52BYM95D type=del 2023-12-03T11:48:39.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.209+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=8G1X53CJEMVL type=del size=1142815 2023-12-03T11:48:39.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.210+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=DO09V1YR0U6L type=del 2023-12-03T11:48:39.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.212+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~YJ3SJQ5QD58X.0 size=1048576 2023-12-03T11:48:39.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.213+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~YJ3SJQ5QD58X.1 size=512941 2023-12-03T11:48:39.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.213+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=YJ3SJQ5QD58X tag=5GC3U6MOE0M6 type=write size=2610093 2023-12-03T11:48:39.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.214+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~JWR3EWYAMPRM.0 size=660978 2023-12-03T11:48:39.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.216+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=JWR3EWYAMPRM tag=8ZL1BL0B4U0X type=write size=1709554 2023-12-03T11:48:39.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.218+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~Z192RJO4IX1B.0 size=583313 2023-12-03T11:48:39.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.218+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=Z192RJO4IX1B tag=BP9P507UEXUF type=write size=1631889 2023-12-03T11:48:39.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.220+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=BCNJ131VTOI7 type=write size=520749 2023-12-03T11:48:39.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.221+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=5Q0H8IMJK894 type=del size=350912 2023-12-03T11:48:39.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.222+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=WC18A08JZZ4J type=del 2023-12-03T11:48:39.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.223+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=3RULY08VQZ84 type=del size=1467669 2023-12-03T11:48:39.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.225+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.0 size=1048576 2023-12-03T11:48:39.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.227+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.1 size=1048576 2023-12-03T11:48:39.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.228+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~4SB5WJZ98O0I.2 size=126390 2023-12-03T11:48:39.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.231+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=4SB5WJZ98O0I tag=UFRWFXEPVH88 type=write size=3272118 2023-12-03T11:48:39.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.232+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=6IZRJEZRO5T6 type=del size=8363 2023-12-03T11:48:39.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.233+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.0 size=1048576 2023-12-03T11:48:39.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.234+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.1 size=1048576 2023-12-03T11:48:39.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.235+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~6WZ6GKAZHXNX.2 size=11260 2023-12-03T11:48:39.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.237+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=6WZ6GKAZHXNX tag=M3Y9J6212P2N type=write size=3156988 2023-12-03T11:48:39.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.238+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=XVRFRETGY51B type=del size=2170972 2023-12-03T11:48:39.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.239+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=YVGDDM5L7MMN type=del 2023-12-03T11:48:39.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.241+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=CODS4SIV3QIZ type=del size=4190281 2023-12-03T11:48:39.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.242+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=2A89MW6OPKCK type=del size=3535253 2023-12-03T11:48:39.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.243+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=TQBE01EN31OO type=del size=591700 2023-12-03T11:48:39.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.244+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=JCGJ29VKK5VP type=del 2023-12-03T11:48:39.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.246+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=58I8LFXWFRQH type=del size=3837635 2023-12-03T11:48:39.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.247+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=CG7B26OF64TY type=del 2023-12-03T11:48:39.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.248+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=740KB3M395LA type=del size=2081209 2023-12-03T11:48:39.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.249+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~VT4NWG6U1A8I.0 size=394896 2023-12-03T11:48:39.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.251+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=VT4NWG6U1A8I tag=D1LGTL4N560O type=write size=1443472 2023-12-03T11:48:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.252+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=0QDTYJJKFMDH type=del size=562753 2023-12-03T11:48:39.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.254+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=RAMD0PK8TPKD type=del 2023-12-03T11:48:39.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.255+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=YY268H1EQ3R2 type=del size=3809703 2023-12-03T11:48:39.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.255+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~1FFFGM6FOFT8.0 size=1048576 2023-12-03T11:48:39.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.255+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~1FFFGM6FOFT8.1 size=617703 2023-12-03T11:48:39.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.257+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=1FFFGM6FOFT8 tag=3E8J7MX4OJGC type=write size=2714855 2023-12-03T11:48:39.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.258+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=VEDMXG5WXQFZ type=del size=34044 2023-12-03T11:48:39.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.260+0000 7f24574f5780 1 < suggested update operation key=PS1OYWI44L80 tag=G21FNBBGOX2I type=del size=1581394 2023-12-03T11:48:39.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.262+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=JOBHP2EAG3JK type=del size=3210032 2023-12-03T11:48:39.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.263+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~19G0DIMEF993.0 size=1048576 2023-12-03T11:48:39.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.264+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~19G0DIMEF993.1 size=248407 2023-12-03T11:48:39.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.266+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=19G0DIMEF993 tag=VBXJWLGWQ6D0 type=write size=2345559 2023-12-03T11:48:39.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.267+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=MR5F1PD2U9EL type=del size=1472474 2023-12-03T11:48:39.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.268+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=8G1X53CJEMVL type=del 2023-12-03T11:48:39.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.269+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=SRJPU2G3FPKY type=write size=705504 2023-12-03T11:48:39.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.270+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~YJ3SJQ5QD58X.0 size=1048576 2023-12-03T11:48:39.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.271+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~YJ3SJQ5QD58X.1 size=512941 2023-12-03T11:48:39.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.273+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=YJ3SJQ5QD58X tag=5GC3U6MOE0M6 type=write size=2610093 2023-12-03T11:48:39.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.275+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=DLMFX447WC41 type=del size=1403623 2023-12-03T11:48:39.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.276+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~Z192RJO4IX1B.0 size=583313 2023-12-03T11:48:39.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.278+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=Z192RJO4IX1B tag=BP9P507UEXUF type=write size=1631889 2023-12-03T11:48:39.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.279+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=KLK4N1Y03ME4 type=del size=3311159 2023-12-03T11:48:39.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.281+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=5Q0H8IMJK894 type=del 2023-12-03T11:48:39.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.283+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=EGZBY4JWWA2S type=del size=3837030 2023-12-03T11:48:39.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.284+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=3RULY08VQZ84 type=del 2023-12-03T11:48:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.285+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=XD5MXWCA8FNG type=del size=1245944 2023-12-03T11:48:39.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.287+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=6IZRJEZRO5T6 type=del 2023-12-03T11:48:39.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.289+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=2KXQFJV6AUQQ type=del size=1032027 2023-12-03T11:48:39.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.291+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=XVRFRETGY51B type=del 2023-12-03T11:48:39.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.293+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.0 size=1048576 2023-12-03T11:48:39.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.293+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.1 size=1048576 2023-12-03T11:48:39.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.294+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.2 size=219940 2023-12-03T11:48:39.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.294+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=Z6JPMSN16E5O tag=B8G3R152VITN type=write size=3365668 2023-12-03T11:48:39.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.298+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=CODS4SIV3QIZ type=del 2023-12-03T11:48:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.299+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=JAFUQKLLT0L4 type=del size=2895661 2023-12-03T11:48:39.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.300+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=TQBE01EN31OO type=del 2023-12-03T11:48:39.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.302+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=94K26H2VW2QF type=del size=4148502 2023-12-03T11:48:39.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.303+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=58I8LFXWFRQH type=del 2023-12-03T11:48:39.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.304+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=0HBVNMEYZK3R type=del size=505508 2023-12-03T11:48:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.305+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=740KB3M395LA type=del 2023-12-03T11:48:39.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.307+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~66CFEPB1CVVC.0 size=365172 2023-12-03T11:48:39.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.307+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=66CFEPB1CVVC tag=XN1NJ5YE9XYR type=write size=1413748 2023-12-03T11:48:39.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.308+0000 7f24574f5780 -1 < failed to complete operation key=QVLPCWC4T8N8 tag=0QDTYJJKFMDH type=del size=562753: (22) Invalid argument 2023-12-03T11:48:39.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.309+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=1NPS0SO3J11S type=write size=180396 2023-12-03T11:48:39.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.310+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=YY268H1EQ3R2 type=del 2023-12-03T11:48:39.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.312+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=M9USP698LCNP type=del size=1493818 2023-12-03T11:48:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.314+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=VEDMXG5WXQFZ type=del size=34044 2023-12-03T11:48:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.315+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=OIACMNJ2XS49 type=del size=3403301 2023-12-03T11:48:39.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.316+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=JOBHP2EAG3JK type=del 2023-12-03T11:48:39.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.317+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=WKW56453GOA9 type=del size=2735088 2023-12-03T11:48:39.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.318+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=MR5F1PD2U9EL type=del 2023-12-03T11:48:39.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.320+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=GYM5T8SIZYH0 type=del size=600995 2023-12-03T11:48:39.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.321+0000 7f24574f5780 -1 < failed to complete operation key=QVLPCWC4T8N8 tag=SRJPU2G3FPKY type=write size=705504: (22) Invalid argument 2023-12-03T11:48:39.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.322+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=VGJPU317COGK type=del size=1933069 2023-12-03T11:48:39.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.323+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=DLMFX447WC41 type=del 2023-12-03T11:48:39.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.324+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=2S73B8VD9CA5 type=del size=3473148 2023-12-03T11:48:39.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.325+0000 7f24574f5780 1 < canceled operation key=U043YAPGUBT5 tag=KLK4N1Y03ME4 type=del size=3311159 2023-12-03T11:48:39.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.326+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=T79DT93BTO57 type=write size=34905 2023-12-03T11:48:39.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.327+0000 7f24574f5780 -1 < failed to complete operation key=QVLPCWC4T8N8 tag=EGZBY4JWWA2S type=del size=3837030: (22) Invalid argument 2023-12-03T11:48:39.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.329+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~2CQVLJ8V0WG9.0 size=382056 2023-12-03T11:48:39.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.329+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=2CQVLJ8V0WG9 tag=KUFW313R27ZT type=write size=1430632 2023-12-03T11:48:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.331+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=XD5MXWCA8FNG type=del 2023-12-03T11:48:39.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.332+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=ASVFEQSPLVXK type=del size=1919394 2023-12-03T11:48:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.334+0000 7f24574f5780 1 < suggested update operation key=9LKFY1U0JRBQ tag=2KXQFJV6AUQQ type=del size=1032027 2023-12-03T11:48:39.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.335+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=PIV37FD2S2W4 type=del size=3240653 2023-12-03T11:48:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.337+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.0 size=1048576 2023-12-03T11:48:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.337+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.1 size=1048576 2023-12-03T11:48:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.338+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~Z6JPMSN16E5O.2 size=219940 2023-12-03T11:48:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.340+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=Z6JPMSN16E5O tag=B8G3R152VITN type=write size=3365668 2023-12-03T11:48:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.341+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=YHUW0OL51XSE type=del size=4093085 2023-12-03T11:48:39.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.342+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=JAFUQKLLT0L4 type=del 2023-12-03T11:48:39.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.344+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~SDK3ALUXABKW.0 size=1048576 2023-12-03T11:48:39.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.345+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~SDK3ALUXABKW.1 size=621581 2023-12-03T11:48:39.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.345+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=SDK3ALUXABKW tag=49I8G4EJFVXN type=write size=2718733 2023-12-03T11:48:39.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.346+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=94K26H2VW2QF type=del 2023-12-03T11:48:39.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.348+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=YUS1MGRXQNW9 type=write size=459627 2023-12-03T11:48:39.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.350+0000 7f24574f5780 1 < suggested update operation key=QVLPCWC4T8N8 tag=0HBVNMEYZK3R type=del size=505508 2023-12-03T11:48:39.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.352+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.0 size=1048576 2023-12-03T11:48:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.353+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.1 size=1048576 2023-12-03T11:48:39.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.354+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.2 size=935408 2023-12-03T11:48:39.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.354+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=W54JN90LAF7U tag=FJ98UL8KAIJB type=write size=4081136 2023-12-03T11:48:39.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.355+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~66CFEPB1CVVC.0 size=365172 2023-12-03T11:48:39.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.357+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=66CFEPB1CVVC tag=XN1NJ5YE9XYR type=write size=1413748 2023-12-03T11:48:39.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.358+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=KKG9J802VUEP type=write size=700541 2023-12-03T11:48:39.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.359+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=1NPS0SO3J11S type=write size=180396 2023-12-03T11:48:39.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.360+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=6326TB3XMWL3 type=write size=826734 2023-12-03T11:48:39.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.361+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=M9USP698LCNP type=del 2023-12-03T11:48:39.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.362+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=VIKJKX0AONR9 type=del size=373159 2023-12-03T11:48:39.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.364+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=OIACMNJ2XS49 type=del 2023-12-03T11:48:39.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.366+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~ZOMQ5US0N5IS.0 size=924823 2023-12-03T11:48:39.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.366+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=ZOMQ5US0N5IS tag=Z1X7L3JFTM5L type=write size=1973399 2023-12-03T11:48:39.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.368+0000 7f24574f5780 1 < suggested update operation key=9LKFY1U0JRBQ tag=WKW56453GOA9 type=del size=2735088 2023-12-03T11:48:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.370+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~YHC0M2K730T2.0 size=131241 2023-12-03T11:48:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.370+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=YHC0M2K730T2 tag=NJDSNO5MKQKV type=write size=1179817 2023-12-03T11:48:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.371+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=GYM5T8SIZYH0 type=del 2023-12-03T11:48:39.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.373+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~BAXIVZM1DFMD.0 size=119468 2023-12-03T11:48:39.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.373+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=BAXIVZM1DFMD tag=M2V5LOE203PK type=write size=1168044 2023-12-03T11:48:39.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.374+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=VGJPU317COGK type=del 2023-12-03T11:48:39.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.376+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~6ATDWYQ0H9HO.0 size=128549 2023-12-03T11:48:39.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.376+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=6ATDWYQ0H9HO tag=4TN2C44R6Y6C type=write size=1177125 2023-12-03T11:48:39.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.378+0000 7f24574f5780 -1 < failed to complete operation key=9LKFY1U0JRBQ tag=2S73B8VD9CA5 type=del size=3473148: (22) Invalid argument 2023-12-03T11:48:39.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.380+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~W17NE5LWCEUL.0 size=1048576 2023-12-03T11:48:39.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.381+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~W17NE5LWCEUL.1 size=420045 2023-12-03T11:48:39.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.381+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=W17NE5LWCEUL tag=4IIOYORAE17T type=write size=2517197 2023-12-03T11:48:39.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.382+0000 7f24574f5780 1 < completed write operation key=1V39QA10M1HY tag=T79DT93BTO57 type=write size=34905 2023-12-03T11:48:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.384+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~7A4U3FY7K6GC.0 size=1014520 2023-12-03T11:48:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.384+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=7A4U3FY7K6GC tag=Q0GYCVEULVIG type=write size=2063096 2023-12-03T11:48:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.385+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~2CQVLJ8V0WG9.0 size=382056 2023-12-03T11:48:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.387+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=2CQVLJ8V0WG9 tag=KUFW313R27ZT type=write size=1430632 2023-12-03T11:48:39.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.388+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=O12R8W3OWNDY type=del size=3615140 2023-12-03T11:48:39.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.389+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=ASVFEQSPLVXK type=del 2023-12-03T11:48:39.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.390+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=R08JQ1A4FYF2 type=write size=457264 2023-12-03T11:48:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.391+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=PIV37FD2S2W4 type=del 2023-12-03T11:48:39.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.394+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.0 size=1048576 2023-12-03T11:48:39.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.394+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.1 size=1048576 2023-12-03T11:48:39.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.395+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.2 size=34239 2023-12-03T11:48:39.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.395+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=GVCNGMZU9ZNG tag=DCI58LBO6RS5 type=write size=3179967 2023-12-03T11:48:39.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.397+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=YHUW0OL51XSE type=del 2023-12-03T11:48:39.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.398+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=LU9N01LAUORJ type=del size=1849261 2023-12-03T11:48:39.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.400+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~SDK3ALUXABKW.0 size=1048576 2023-12-03T11:48:39.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.400+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~SDK3ALUXABKW.1 size=621581 2023-12-03T11:48:39.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.402+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=SDK3ALUXABKW tag=49I8G4EJFVXN type=write size=2718733 2023-12-03T11:48:39.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.405+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~L9703Z463357.0 size=1048576 2023-12-03T11:48:39.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.405+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~L9703Z463357.1 size=572237 2023-12-03T11:48:39.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.405+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=L9703Z463357 tag=3D4YSM2Y68FG type=write size=2669389 2023-12-03T11:48:39.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.407+0000 7f24574f5780 -1 < failed to complete operation key=9LKFY1U0JRBQ tag=YUS1MGRXQNW9 type=write size=459627: (22) Invalid argument 2023-12-03T11:48:39.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.408+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=K7JOWN4AZQLX type=write size=897344 2023-12-03T11:48:39.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.409+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.0 size=1048576 2023-12-03T11:48:39.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.410+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.1 size=1048576 2023-12-03T11:48:39.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.411+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~W54JN90LAF7U.2 size=935408 2023-12-03T11:48:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.413+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=W54JN90LAF7U tag=FJ98UL8KAIJB type=write size=4081136 2023-12-03T11:48:39.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.414+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=O4ZP0HP0FLDN type=del size=1503779 2023-12-03T11:48:39.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.415+0000 7f24574f5780 1 < canceled operation key=S73M6UFMX5NH tag=KKG9J802VUEP type=write size=700541 2023-12-03T11:48:39.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.416+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=TW5UIHCNB7KL type=write size=17336 2023-12-03T11:48:39.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.417+0000 7f24574f5780 1 < completed write operation key=1QR0TE9DXU5Z tag=6326TB3XMWL3 type=write size=826734 2023-12-03T11:48:39.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.419+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.0 size=1048576 2023-12-03T11:48:39.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.420+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.1 size=1048576 2023-12-03T11:48:39.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.421+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.2 size=795983 2023-12-03T11:48:39.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.421+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=3AXWQIT6HQ8E tag=X1LV90JAP29E type=write size=3941711 2023-12-03T11:48:39.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.422+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=VIKJKX0AONR9 type=del 2023-12-03T11:48:39.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.423+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=3T79ZPIEKJD3 type=del size=3676413 2023-12-03T11:48:39.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.425+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~ZOMQ5US0N5IS.0 size=924823 2023-12-03T11:48:39.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.426+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=ZOMQ5US0N5IS tag=Z1X7L3JFTM5L type=write size=1973399 2023-12-03T11:48:39.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.429+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~RZ6H3HIFNR74.0 size=1048576 2023-12-03T11:48:39.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.429+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~RZ6H3HIFNR74.1 size=350683 2023-12-03T11:48:39.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.429+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=RZ6H3HIFNR74 tag=ZA4Z116SG4ED type=write size=2447835 2023-12-03T11:48:39.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.431+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~YHC0M2K730T2.0 size=131241 2023-12-03T11:48:39.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.432+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=YHC0M2K730T2 tag=NJDSNO5MKQKV type=write size=1179817 2023-12-03T11:48:39.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.434+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=XX4P5Z9AMLA3 type=del size=2639414 2023-12-03T11:48:39.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.435+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~BAXIVZM1DFMD.0 size=119468 2023-12-03T11:48:39.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.437+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=BAXIVZM1DFMD tag=M2V5LOE203PK type=write size=1168044 2023-12-03T11:48:39.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.438+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=TOQDA66B66DU type=del size=3185076 2023-12-03T11:48:39.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.439+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~6ATDWYQ0H9HO.0 size=128549 2023-12-03T11:48:39.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.441+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=6ATDWYQ0H9HO tag=4TN2C44R6Y6C type=write size=1177125 2023-12-03T11:48:39.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.442+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=0QG62D9D078M type=del size=3953259 2023-12-03T11:48:39.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.443+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~W17NE5LWCEUL.0 size=1048576 2023-12-03T11:48:39.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.443+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~W17NE5LWCEUL.1 size=420045 2023-12-03T11:48:39.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.445+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=W17NE5LWCEUL tag=4IIOYORAE17T type=write size=2517197 2023-12-03T11:48:39.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.447+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.0 size=1048576 2023-12-03T11:48:39.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.448+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.1 size=1048576 2023-12-03T11:48:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.449+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.2 size=277958 2023-12-03T11:48:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.449+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=YPMTCL6R7WC6 tag=0SNKGH9NH54V type=write size=3423686 2023-12-03T11:48:39.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.450+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~7A4U3FY7K6GC.0 size=1014520 2023-12-03T11:48:39.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.452+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=7A4U3FY7K6GC tag=Q0GYCVEULVIG type=write size=2063096 2023-12-03T11:48:39.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.453+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=OIRXBGIAU0HV type=del size=505227 2023-12-03T11:48:39.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.454+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=O12R8W3OWNDY type=del 2023-12-03T11:48:39.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.456+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=9VNFYI5MZJC4 type=del size=1538698 2023-12-03T11:48:39.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.457+0000 7f24574f5780 1 < completed write operation key=3EOEF9UMATIE tag=R08JQ1A4FYF2 type=write size=457264 2023-12-03T11:48:39.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.459+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~9T2P10S1WDV7.0 size=1048576 2023-12-03T11:48:39.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.460+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~9T2P10S1WDV7.1 size=94002 2023-12-03T11:48:39.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.460+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=9T2P10S1WDV7 tag=J01LSH1QI8VL type=write size=2191154 2023-12-03T11:48:39.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.460+0000 7f24574f5780 1 < canceled part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.0 size=1048576 2023-12-03T11:48:39.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.461+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.1 size=1048576 2023-12-03T11:48:39.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.462+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~GVCNGMZU9ZNG.2 size=34239 2023-12-03T11:48:39.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.464+0000 7f24574f5780 1 < canceled multipart upload key=MH1SRAATMNXB upload=GVCNGMZU9ZNG tag=DCI58LBO6RS5 type=write size=3179967 2023-12-03T11:48:39.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.465+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=BKD4CO2CXPYA type=del size=3281639 2023-12-03T11:48:39.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.467+0000 7f24574f5780 1 < suggested update operation key=XO42YZR60KQE tag=LU9N01LAUORJ type=del size=1849261 2023-12-03T11:48:39.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.468+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=SYTSJLLGEUD9 type=del size=810708 2023-12-03T11:48:39.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.469+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~L9703Z463357.0 size=1048576 2023-12-03T11:48:39.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.470+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~L9703Z463357.1 size=572237 2023-12-03T11:48:39.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.472+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=L9703Z463357 tag=3D4YSM2Y68FG type=write size=2669389 2023-12-03T11:48:39.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.473+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=1IPCQUOHWC1K type=write size=158649 2023-12-03T11:48:39.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.474+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=K7JOWN4AZQLX type=write size=897344 2023-12-03T11:48:39.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.475+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=SZXWO71ISJWC type=del size=22788 2023-12-03T11:48:39.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.478+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=O4ZP0HP0FLDN type=del size=1503779 2023-12-03T11:48:39.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.480+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~3LQ1WPKF6NNV.0 size=1048576 2023-12-03T11:48:39.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.480+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~3LQ1WPKF6NNV.1 size=284378 2023-12-03T11:48:39.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.480+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=3LQ1WPKF6NNV tag=MTRIN42PPQGK type=write size=2381530 2023-12-03T11:48:39.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.482+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=TW5UIHCNB7KL type=write size=17336 2023-12-03T11:48:39.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.484+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~WYXKI23X02S7.0 size=1048576 2023-12-03T11:48:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.485+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~WYXKI23X02S7.1 size=912281 2023-12-03T11:48:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.485+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=WYXKI23X02S7 tag=4XR0W6YWQ2LC type=write size=3009433 2023-12-03T11:48:39.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.486+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.0 size=1048576 2023-12-03T11:48:39.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.487+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.1 size=1048576 2023-12-03T11:48:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.488+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3AXWQIT6HQ8E.2 size=795983 2023-12-03T11:48:39.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.490+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=3AXWQIT6HQ8E tag=X1LV90JAP29E type=write size=3941711 2023-12-03T11:48:39.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.492+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~GOSONPPG2BVY.0 size=468888 2023-12-03T11:48:39.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.492+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=GOSONPPG2BVY tag=GBCGWQIJ1W8U type=write size=1517464 2023-12-03T11:48:39.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.493+0000 7f24574f5780 -1 < failed to complete operation key=XRHVYVADRWET tag=3T79ZPIEKJD3 type=del size=3676413: (22) Invalid argument 2023-12-03T11:48:39.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.494+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=5KHUI9ESVXY8 type=del size=286649 2023-12-03T11:48:39.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.496+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~RZ6H3HIFNR74.0 size=1048576 2023-12-03T11:48:39.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.497+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~RZ6H3HIFNR74.1 size=350683 2023-12-03T11:48:39.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.499+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=RZ6H3HIFNR74 tag=ZA4Z116SG4ED type=write size=2447835 2023-12-03T11:48:39.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.500+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=CGNQDUMU3ROE type=del size=3135542 2023-12-03T11:48:39.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.501+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=XX4P5Z9AMLA3 type=del 2023-12-03T11:48:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.504+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~LVNNYAO1GDQ7.0 size=1048576 2023-12-03T11:48:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.507+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~LVNNYAO1GDQ7.1 size=457844 2023-12-03T11:48:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.507+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=LVNNYAO1GDQ7 tag=1LHTW11TNZ4I type=write size=2554996 2023-12-03T11:48:39.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.509+0000 7f24574f5780 1 < suggested update operation key=1QR0TE9DXU5Z tag=TOQDA66B66DU type=del size=3185076 2023-12-03T11:48:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.510+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=CQ1SF14LKQTN type=del size=2660590 2023-12-03T11:48:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.512+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=0QG62D9D078M type=del 2023-12-03T11:48:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.513+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=KH8BX8X30DRH type=del size=129380 2023-12-03T11:48:39.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.514+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.0 size=1048576 2023-12-03T11:48:39.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.514+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.1 size=1048576 2023-12-03T11:48:39.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.515+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~YPMTCL6R7WC6.2 size=277958 2023-12-03T11:48:39.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.517+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=YPMTCL6R7WC6 tag=0SNKGH9NH54V type=write size=3423686 2023-12-03T11:48:39.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.519+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=L9OV71K097DS type=del size=2505101 2023-12-03T11:48:39.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.520+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=OIRXBGIAU0HV type=del 2023-12-03T11:48:39.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.521+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=BTZZAQ0N1P88 type=del size=2731709 2023-12-03T11:48:39.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.522+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=9VNFYI5MZJC4 type=del 2023-12-03T11:48:39.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.525+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~9NLIJD713DBI.0 size=202442 2023-12-03T11:48:39.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.525+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=9NLIJD713DBI tag=QI7FIWW7Y4U7 type=write size=1251018 2023-12-03T11:48:39.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.526+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~9T2P10S1WDV7.0 size=1048576 2023-12-03T11:48:39.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.526+0000 7f24574f5780 1 < canceled part key=_multipart_XZFHFS76WNR4.2~9T2P10S1WDV7.1 size=94002 2023-12-03T11:48:39.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.528+0000 7f24574f5780 1 < canceled multipart upload key=XZFHFS76WNR4 upload=9T2P10S1WDV7 tag=J01LSH1QI8VL type=write size=2191154 2023-12-03T11:48:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.530+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.0 size=1048576 2023-12-03T11:48:39.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.531+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.1 size=1048576 2023-12-03T11:48:39.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.532+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.2 size=827293 2023-12-03T11:48:39.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.532+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=PPCQHOGNVK2E tag=L9HD01TUC8XE type=write size=3973021 2023-12-03T11:48:39.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.534+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=BKD4CO2CXPYA type=del 2023-12-03T11:48:39.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.535+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=X6VG861YWF1Y type=del size=1952951 2023-12-03T11:48:39.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.536+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=SYTSJLLGEUD9 type=del 2023-12-03T11:48:39.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.537+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=90766W8YON13 type=write size=691104 2023-12-03T11:48:39.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.539+0000 7f24574f5780 1 < completed write operation key=35U7YCDYMX46 tag=1IPCQUOHWC1K type=write size=158649 2023-12-03T11:48:39.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.540+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=3OZ59GCO2KBO type=write size=364153 2023-12-03T11:48:39.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.541+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=SZXWO71ISJWC type=del 2023-12-03T11:48:39.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.543+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~OJZZPA1QDTPD.0 size=80833 2023-12-03T11:48:39.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.543+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=OJZZPA1QDTPD tag=XDDDKS5NKXOI type=write size=1129409 2023-12-03T11:48:39.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.544+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~3LQ1WPKF6NNV.0 size=1048576 2023-12-03T11:48:39.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.545+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~3LQ1WPKF6NNV.1 size=284378 2023-12-03T11:48:39.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.547+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=3LQ1WPKF6NNV tag=MTRIN42PPQGK type=write size=2381530 2023-12-03T11:48:39.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.549+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~XKV9RDX8EGOE.0 size=1048576 2023-12-03T11:48:39.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.550+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~XKV9RDX8EGOE.1 size=458792 2023-12-03T11:48:39.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.550+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=XKV9RDX8EGOE tag=11LJPBPNZDG7 type=write size=2555944 2023-12-03T11:48:39.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.551+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~WYXKI23X02S7.0 size=1048576 2023-12-03T11:48:39.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.552+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~WYXKI23X02S7.1 size=912281 2023-12-03T11:48:39.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.554+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=WYXKI23X02S7 tag=4XR0W6YWQ2LC type=write size=3009433 2023-12-03T11:48:39.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.556+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.0 size=1048576 2023-12-03T11:48:39.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.557+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.1 size=1048576 2023-12-03T11:48:39.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.558+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.2 size=43976 2023-12-03T11:48:39.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.558+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=L2NUAY7UKBBD tag=05KRQM07RTWT type=write size=3189704 2023-12-03T11:48:39.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.559+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~GOSONPPG2BVY.0 size=468888 2023-12-03T11:48:39.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.561+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=GOSONPPG2BVY tag=GBCGWQIJ1W8U type=write size=1517464 2023-12-03T11:48:39.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.562+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=4F8530YBGP9I type=del size=3978098 2023-12-03T11:48:39.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.563+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=5KHUI9ESVXY8 type=del 2023-12-03T11:48:39.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.565+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=GH9S7NOUGNGD type=del size=3339915 2023-12-03T11:48:39.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.567+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=CGNQDUMU3ROE type=del 2023-12-03T11:48:39.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.568+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=VX1AXHKLGWNS type=write size=19911 2023-12-03T11:48:39.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.569+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~LVNNYAO1GDQ7.0 size=1048576 2023-12-03T11:48:39.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.570+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~LVNNYAO1GDQ7.1 size=457844 2023-12-03T11:48:39.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.572+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=LVNNYAO1GDQ7 tag=1LHTW11TNZ4I type=write size=2554996 2023-12-03T11:48:39.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.574+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UYC5XLTIBXUQ.0 size=541350 2023-12-03T11:48:39.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.574+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=UYC5XLTIBXUQ tag=MRPK63EOEQIF type=write size=1589926 2023-12-03T11:48:39.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.575+0000 7f24574f5780 -1 < failed to complete operation key=S0EH0Q0UA8HG tag=CQ1SF14LKQTN type=del size=2660590: (22) Invalid argument 2023-12-03T11:48:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.576+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=OMMK4G62BQ20 type=del size=3171964 2023-12-03T11:48:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.578+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=KH8BX8X30DRH type=del 2023-12-03T11:48:39.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.579+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=NM4TM9M7WYNO type=write size=140891 2023-12-03T11:48:39.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.580+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=L9OV71K097DS type=del 2023-12-03T11:48:39.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.581+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=CD8MKO85OB1O type=del size=2465608 2023-12-03T11:48:39.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.582+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=BTZZAQ0N1P88 type=del 2023-12-03T11:48:39.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.584+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~PKZBE8BR0JBH.0 size=807999 2023-12-03T11:48:39.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.584+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=PKZBE8BR0JBH tag=VCCM9TNCK2HI type=write size=1856575 2023-12-03T11:48:39.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.586+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~9NLIJD713DBI.0 size=202442 2023-12-03T11:48:39.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.587+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=9NLIJD713DBI tag=QI7FIWW7Y4U7 type=write size=1251018 2023-12-03T11:48:39.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.589+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=NV7INHTJLCX7 type=write size=420554 2023-12-03T11:48:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.590+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.0 size=1048576 2023-12-03T11:48:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.591+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.1 size=1048576 2023-12-03T11:48:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.592+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~PPCQHOGNVK2E.2 size=827293 2023-12-03T11:48:39.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.594+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=PPCQHOGNVK2E tag=L9HD01TUC8XE type=write size=3973021 2023-12-03T11:48:39.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.595+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=1XFA5ZW8FNEM type=del size=266025 2023-12-03T11:48:39.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.596+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=X6VG861YWF1Y type=del 2023-12-03T11:48:39.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.598+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=19MN82RDOX8V type=del size=3175116 2023-12-03T11:48:39.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.600+0000 7f24574f5780 1 < suggested update operation key=DA6ZXCLKTC1L tag=90766W8YON13 type=write size=691104 2023-12-03T11:48:39.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.603+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.0 size=1048576 2023-12-03T11:48:39.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.604+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.1 size=1048576 2023-12-03T11:48:39.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.605+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.2 size=446177 2023-12-03T11:48:39.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.605+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=7GPO407KWLDR tag=G2101UISAIF8 type=write size=3591905 2023-12-03T11:48:39.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.606+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=3OZ59GCO2KBO type=write size=364153 2023-12-03T11:48:39.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.607+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=QNHEW9KJIRCC type=del size=1363455 2023-12-03T11:48:39.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.608+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~OJZZPA1QDTPD.0 size=80833 2023-12-03T11:48:39.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.610+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=OJZZPA1QDTPD tag=XDDDKS5NKXOI type=write size=1129409 2023-12-03T11:48:39.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.612+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~33TC2FN6XBS4.0 size=1048576 2023-12-03T11:48:39.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.613+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~33TC2FN6XBS4.1 size=998399 2023-12-03T11:48:39.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.613+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=33TC2FN6XBS4 tag=UH3SA85R5WON type=write size=3095551 2023-12-03T11:48:39.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.615+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~XKV9RDX8EGOE.0 size=1048576 2023-12-03T11:48:39.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.615+0000 7f24574f5780 1 < canceled part key=_multipart_LS4OEAGZWQXC.2~XKV9RDX8EGOE.1 size=458792 2023-12-03T11:48:39.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.618+0000 7f24574f5780 1 < canceled multipart upload key=LS4OEAGZWQXC upload=XKV9RDX8EGOE tag=11LJPBPNZDG7 type=write size=2555944 2023-12-03T11:48:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.620+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=129PJVQ7KUNE type=del size=992189 2023-12-03T11:48:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.620+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.0 size=1048576 2023-12-03T11:48:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.621+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.1 size=1048576 2023-12-03T11:48:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.622+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~L2NUAY7UKBBD.2 size=43976 2023-12-03T11:48:39.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.624+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=L2NUAY7UKBBD tag=05KRQM07RTWT type=write size=3189704 2023-12-03T11:48:39.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.625+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=P7PMJ7MUMICC type=del size=3637903 2023-12-03T11:48:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.627+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=4F8530YBGP9I type=del 2023-12-03T11:48:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.628+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=YPG283ZNKIYG type=del size=2903821 2023-12-03T11:48:39.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.630+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=GH9S7NOUGNGD type=del size=3339915 2023-12-03T11:48:39.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.631+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=BZ162O5OTQB6 type=del size=3282578 2023-12-03T11:48:39.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.632+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=VX1AXHKLGWNS type=write size=19911 2023-12-03T11:48:39.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.633+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=BPLUKRC1GM99 type=del size=2021642 2023-12-03T11:48:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.635+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UYC5XLTIBXUQ.0 size=541350 2023-12-03T11:48:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.636+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=UYC5XLTIBXUQ tag=MRPK63EOEQIF type=write size=1589926 2023-12-03T11:48:39.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.638+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=PW2M360B4VE0 type=del size=739880 2023-12-03T11:48:39.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.639+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=OMMK4G62BQ20 type=del 2023-12-03T11:48:39.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.641+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.0 size=1048576 2023-12-03T11:48:39.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.642+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.1 size=1048576 2023-12-03T11:48:39.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.643+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.2 size=486941 2023-12-03T11:48:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.643+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=78X5K78JTKJC tag=41RR3EOCAJG8 type=write size=3632669 2023-12-03T11:48:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.644+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=NM4TM9M7WYNO type=write size=140891 2023-12-03T11:48:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.647+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~8FTCEU3T9CP2.0 size=601888 2023-12-03T11:48:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.647+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=8FTCEU3T9CP2 tag=EVQ4EQ6L9R9V type=write size=1650464 2023-12-03T11:48:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.648+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=CD8MKO85OB1O type=del 2023-12-03T11:48:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.650+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~RHT4JMBZQ69D.0 size=1039777 2023-12-03T11:48:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.650+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=RHT4JMBZQ69D tag=R3RX7XU7BTEI type=write size=2088353 2023-12-03T11:48:39.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.651+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~PKZBE8BR0JBH.0 size=807999 2023-12-03T11:48:39.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.653+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=PKZBE8BR0JBH tag=VCCM9TNCK2HI type=write size=1856575 2023-12-03T11:48:39.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.655+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.0 size=1048576 2023-12-03T11:48:39.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.655+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.1 size=1048576 2023-12-03T11:48:39.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.656+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.2 size=980488 2023-12-03T11:48:39.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.656+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=BHBAMP8QRK0Y tag=2YOUPGIQJZA9 type=write size=4126216 2023-12-03T11:48:39.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.657+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=NV7INHTJLCX7 type=write size=420554 2023-12-03T11:48:39.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.659+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.0 size=1048576 2023-12-03T11:48:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.660+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.1 size=1048576 2023-12-03T11:48:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.661+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.2 size=538164 2023-12-03T11:48:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.661+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=6ZJW9SNN0UFQ tag=BTOQHI7KMPTN type=write size=3683892 2023-12-03T11:48:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.662+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=1XFA5ZW8FNEM type=del 2023-12-03T11:48:39.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.664+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=QRQQ272KJ34A type=del size=2603766 2023-12-03T11:48:39.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.666+0000 7f24574f5780 1 < suggested update operation key=XO42YZR60KQE tag=19MN82RDOX8V type=del size=3175116 2023-12-03T11:48:39.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.667+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=ZBZ5FIEUAW46 type=del size=3854679 2023-12-03T11:48:39.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.668+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.0 size=1048576 2023-12-03T11:48:39.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.668+0000 7f24574f5780 1 < canceled part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.1 size=1048576 2023-12-03T11:48:39.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.669+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~7GPO407KWLDR.2 size=446177 2023-12-03T11:48:39.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.671+0000 7f24574f5780 1 < canceled multipart upload key=PS1OYWI44L80 upload=7GPO407KWLDR tag=G2101UISAIF8 type=write size=3591905 2023-12-03T11:48:39.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.672+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=599R13VRXNVO type=del size=1891461 2023-12-03T11:48:39.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.673+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=QNHEW9KJIRCC type=del 2023-12-03T11:48:39.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.674+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=IURNUNGPJ8UJ type=del size=603170 2023-12-03T11:48:39.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.675+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~33TC2FN6XBS4.0 size=1048576 2023-12-03T11:48:39.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.676+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~33TC2FN6XBS4.1 size=998399 2023-12-03T11:48:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.678+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=33TC2FN6XBS4 tag=UH3SA85R5WON type=write size=3095551 2023-12-03T11:48:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.680+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~563N7X7TJE3B.0 size=1048576 2023-12-03T11:48:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.681+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~563N7X7TJE3B.1 size=488981 2023-12-03T11:48:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.681+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=563N7X7TJE3B tag=4NOJXTJEICY0 type=write size=2586133 2023-12-03T11:48:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.682+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=129PJVQ7KUNE type=del 2023-12-03T11:48:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.683+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=Q286WKA3L0PL type=del size=4082727 2023-12-03T11:48:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.684+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=P7PMJ7MUMICC type=del 2023-12-03T11:48:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.687+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.0 size=1048576 2023-12-03T11:48:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.688+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.1 size=1048576 2023-12-03T11:48:39.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.689+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.2 size=627654 2023-12-03T11:48:39.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.689+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=NDS3JJLOJOWL tag=W3ZK9NGPYZYF type=write size=3773382 2023-12-03T11:48:39.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.690+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=YPG283ZNKIYG type=del 2023-12-03T11:48:39.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.691+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=TO8THYTO6VHZ type=del size=1541678 2023-12-03T11:48:39.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.692+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=BZ162O5OTQB6 type=del 2023-12-03T11:48:39.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.694+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=JWDV6641CBMN type=del size=640328 2023-12-03T11:48:39.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.695+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=BPLUKRC1GM99 type=del 2023-12-03T11:48:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.696+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=DY4K0GXLRX3I type=del size=928809 2023-12-03T11:48:39.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.697+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=PW2M360B4VE0 type=del 2023-12-03T11:48:39.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.699+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=PRHQP28HSAPK type=del size=574582 2023-12-03T11:48:39.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.700+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.0 size=1048576 2023-12-03T11:48:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.701+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.1 size=1048576 2023-12-03T11:48:39.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.702+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~78X5K78JTKJC.2 size=486941 2023-12-03T11:48:39.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.704+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=78X5K78JTKJC tag=41RR3EOCAJG8 type=write size=3632669 2023-12-03T11:48:39.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.705+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=2Y3VPQ583GDI type=write size=758158 2023-12-03T11:48:39.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.705+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~8FTCEU3T9CP2.0 size=601888 2023-12-03T11:48:39.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.707+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=8FTCEU3T9CP2 tag=EVQ4EQ6L9R9V type=write size=1650464 2023-12-03T11:48:39.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.709+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~OUFGUG3O1GKI.0 size=1048576 2023-12-03T11:48:39.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.710+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~OUFGUG3O1GKI.1 size=238128 2023-12-03T11:48:39.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.710+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=OUFGUG3O1GKI tag=DQ9JFEJTYEA0 type=write size=2335280 2023-12-03T11:48:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.711+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~RHT4JMBZQ69D.0 size=1039777 2023-12-03T11:48:39.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.713+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=RHT4JMBZQ69D tag=R3RX7XU7BTEI type=write size=2088353 2023-12-03T11:48:39.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.716+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.0 size=1048576 2023-12-03T11:48:39.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.716+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.1 size=1048576 2023-12-03T11:48:39.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.717+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.2 size=284173 2023-12-03T11:48:39.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.717+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=I9LSFXET2YOE tag=SHUTHQXF0TRV type=write size=3429901 2023-12-03T11:48:39.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.719+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.0 size=1048576 2023-12-03T11:48:39.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.720+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.1 size=1048576 2023-12-03T11:48:39.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.721+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~BHBAMP8QRK0Y.2 size=980488 2023-12-03T11:48:39.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.723+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=BHBAMP8QRK0Y tag=2YOUPGIQJZA9 type=write size=4126216 2023-12-03T11:48:39.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.724+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=D0ZRMLXAFXEW type=del size=2884575 2023-12-03T11:48:39.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.726+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.0 size=1048576 2023-12-03T11:48:39.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.727+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.1 size=1048576 2023-12-03T11:48:39.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.727+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~6ZJW9SNN0UFQ.2 size=538164 2023-12-03T11:48:39.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.731+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=6ZJW9SNN0UFQ tag=BTOQHI7KMPTN type=write size=3683892 2023-12-03T11:48:39.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.732+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=B1RL8H1JJDX0 type=del size=1394713 2023-12-03T11:48:39.734 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.733+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=QRQQ272KJ34A type=del 2023-12-03T11:48:39.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.736+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=EU84KKKVTRV8 type=del size=667451 2023-12-03T11:48:39.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.737+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=ZBZ5FIEUAW46 type=del 2023-12-03T11:48:39.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.739+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~JNNDKQRR53WN.0 size=1048576 2023-12-03T11:48:39.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.740+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~JNNDKQRR53WN.1 size=635508 2023-12-03T11:48:39.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.740+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=JNNDKQRR53WN tag=DWGEDP74EX1Q type=write size=2732660 2023-12-03T11:48:39.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.741+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=599R13VRXNVO type=del 2023-12-03T11:48:39.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.743+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.0 size=1048576 2023-12-03T11:48:39.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.744+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.1 size=1048576 2023-12-03T11:48:39.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.745+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.2 size=672915 2023-12-03T11:48:39.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.745+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=UTHSFMVYPH6Z tag=7KVOVRBITI0A type=write size=3818643 2023-12-03T11:48:39.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.746+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=IURNUNGPJ8UJ type=del 2023-12-03T11:48:39.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.747+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=D2TRLSV8N7XS type=del size=551568 2023-12-03T11:48:39.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.749+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~563N7X7TJE3B.0 size=1048576 2023-12-03T11:48:39.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.750+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~563N7X7TJE3B.1 size=488981 2023-12-03T11:48:39.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.752+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=563N7X7TJE3B tag=4NOJXTJEICY0 type=write size=2586133 2023-12-03T11:48:39.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.753+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=OKD2Q8YU5WGI type=del size=4032436 2023-12-03T11:48:39.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.754+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=Q286WKA3L0PL type=del 2023-12-03T11:48:39.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.755+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=O1F8WRYIRY8Z type=write size=312747 2023-12-03T11:48:39.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.756+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.0 size=1048576 2023-12-03T11:48:39.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.758+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.1 size=1048576 2023-12-03T11:48:39.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.758+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~NDS3JJLOJOWL.2 size=627654 2023-12-03T11:48:39.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.759+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=NDS3JJLOJOWL tag=W3ZK9NGPYZYF type=write size=3773382 2023-12-03T11:48:39.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.762+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.0 size=1048576 2023-12-03T11:48:39.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.762+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.1 size=1048576 2023-12-03T11:48:39.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.763+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.2 size=357280 2023-12-03T11:48:39.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.763+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=RA4Z1FRT6OF3 tag=UP21U00ZOR5J type=write size=3503008 2023-12-03T11:48:39.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.765+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=TO8THYTO6VHZ type=del 2023-12-03T11:48:39.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.767+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=9SIVTDMAGB0F type=del size=1744356 2023-12-03T11:48:39.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.769+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=JWDV6641CBMN type=del size=640328 2023-12-03T11:48:39.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.771+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=PAZA6GZG5C8X type=del size=2755195 2023-12-03T11:48:39.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.772+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=DY4K0GXLRX3I type=del 2023-12-03T11:48:39.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.774+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~3OAE0JTYKS7D.0 size=399433 2023-12-03T11:48:39.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.774+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=3OAE0JTYKS7D tag=4OOPA62CQIUR type=write size=1448009 2023-12-03T11:48:39.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.775+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=PRHQP28HSAPK type=del 2023-12-03T11:48:39.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.777+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~M5LJU97VSGF1.0 size=1042380 2023-12-03T11:48:39.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.777+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=M5LJU97VSGF1 tag=O90Y4IY8M1XN type=write size=2090956 2023-12-03T11:48:39.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.778+0000 7f24574f5780 1 < completed write operation key=WD7U31FW3NGZ tag=2Y3VPQ583GDI type=write size=758158 2023-12-03T11:48:39.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.781+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~M9RPPRMNG2JS.0 size=1048576 2023-12-03T11:48:39.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.781+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~M9RPPRMNG2JS.1 size=479354 2023-12-03T11:48:39.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.781+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=M9RPPRMNG2JS tag=91F0KQOX905G type=write size=2576506 2023-12-03T11:48:39.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.783+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~OUFGUG3O1GKI.0 size=1048576 2023-12-03T11:48:39.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.784+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~OUFGUG3O1GKI.1 size=238128 2023-12-03T11:48:39.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.786+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=OUFGUG3O1GKI tag=DQ9JFEJTYEA0 type=write size=2335280 2023-12-03T11:48:39.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.788+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~94ZYC9MDU2TE.0 size=363785 2023-12-03T11:48:39.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.788+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=94ZYC9MDU2TE tag=Y6QLHE7NHKJO type=write size=1412361 2023-12-03T11:48:39.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.789+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.0 size=1048576 2023-12-03T11:48:39.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.790+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.1 size=1048576 2023-12-03T11:48:39.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.791+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~I9LSFXET2YOE.2 size=284173 2023-12-03T11:48:39.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.793+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=I9LSFXET2YOE tag=SHUTHQXF0TRV type=write size=3429901 2023-12-03T11:48:39.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.796+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~1GM36KE3QNNF.0 size=1048576 2023-12-03T11:48:39.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.797+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~1GM36KE3QNNF.1 size=95808 2023-12-03T11:48:39.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.797+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=1GM36KE3QNNF tag=DMB5A1MDIX80 type=write size=2192960 2023-12-03T11:48:39.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.799+0000 7f24574f5780 1 < suggested remove operation key=XRHVYVADRWET tag=D0ZRMLXAFXEW type=del 2023-12-03T11:48:39.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.800+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=V5C10TIX2T63 type=del size=1925926 2023-12-03T11:48:39.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.801+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=B1RL8H1JJDX0 type=del 2023-12-03T11:48:39.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.803+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~RDAZHL0D42SE.0 size=1048576 2023-12-03T11:48:39.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.804+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~RDAZHL0D42SE.1 size=405121 2023-12-03T11:48:39.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.804+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=RDAZHL0D42SE tag=71HYLL0435O3 type=write size=2502273 2023-12-03T11:48:39.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.806+0000 7f24574f5780 1 < suggested remove operation key=6WY91DFIRUN8 tag=EU84KKKVTRV8 type=del 2023-12-03T11:48:39.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.808+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=8RP1QVXU4V5I type=del size=4193698 2023-12-03T11:48:39.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.809+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~JNNDKQRR53WN.0 size=1048576 2023-12-03T11:48:39.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.810+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~JNNDKQRR53WN.1 size=635508 2023-12-03T11:48:39.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.812+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=JNNDKQRR53WN tag=DWGEDP74EX1Q type=write size=2732660 2023-12-03T11:48:39.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.814+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~THCX3VWUNGXI.0 size=664158 2023-12-03T11:48:39.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.814+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=THCX3VWUNGXI tag=R69YRCHSMGIM type=write size=1712734 2023-12-03T11:48:39.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.815+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.0 size=1048576 2023-12-03T11:48:39.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.816+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.1 size=1048576 2023-12-03T11:48:39.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.817+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~UTHSFMVYPH6Z.2 size=672915 2023-12-03T11:48:39.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.819+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=UTHSFMVYPH6Z tag=7KVOVRBITI0A type=write size=3818643 2023-12-03T11:48:39.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.820+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=2GR0HO71ZH0E type=write size=391155 2023-12-03T11:48:39.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.821+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=D2TRLSV8N7XS type=del 2023-12-03T11:48:39.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.823+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~SRWUITPFOA1S.0 size=60120 2023-12-03T11:48:39.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.823+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=SRWUITPFOA1S tag=3MG6M40VTR5G type=write size=1108696 2023-12-03T11:48:39.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.825+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=OKD2Q8YU5WGI type=del 2023-12-03T11:48:39.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.826+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=2WK76ANBH9W4 type=del size=1580636 2023-12-03T11:48:39.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.827+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=O1F8WRYIRY8Z type=write size=312747 2023-12-03T11:48:39.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.828+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=YIBE2S00TVBC type=write size=723565 2023-12-03T11:48:39.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.830+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.0 size=1048576 2023-12-03T11:48:39.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.831+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.1 size=1048576 2023-12-03T11:48:39.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.832+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~RA4Z1FRT6OF3.2 size=357280 2023-12-03T11:48:39.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.834+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=RA4Z1FRT6OF3 tag=UP21U00ZOR5J type=write size=3503008 2023-12-03T11:48:39.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.836+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.0 size=1048576 2023-12-03T11:48:39.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.837+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.1 size=1048576 2023-12-03T11:48:39.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.838+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.2 size=825100 2023-12-03T11:48:39.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.838+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=Q3O9L21RD1DO tag=T1RZVCQXAPH7 type=write size=3970828 2023-12-03T11:48:39.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.840+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=9SIVTDMAGB0F type=del 2023-12-03T11:48:39.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.841+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=2VN2D4UVRJYU type=del size=55881 2023-12-03T11:48:39.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.842+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=PAZA6GZG5C8X type=del 2023-12-03T11:48:39.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.844+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~WKAXL5G54VYG.0 size=45913 2023-12-03T11:48:39.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.844+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=WKAXL5G54VYG tag=I2N4GIDGUMNZ type=write size=1094489 2023-12-03T11:48:39.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.845+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~3OAE0JTYKS7D.0 size=399433 2023-12-03T11:48:39.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.847+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=3OAE0JTYKS7D tag=4OOPA62CQIUR type=write size=1448009 2023-12-03T11:48:39.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.848+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=3UMHM8BZE6F5 type=del size=809384 2023-12-03T11:48:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.850+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~M5LJU97VSGF1.0 size=1042380 2023-12-03T11:48:39.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.851+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=M5LJU97VSGF1 tag=O90Y4IY8M1XN type=write size=2090956 2023-12-03T11:48:39.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.852+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=1ZGO2OO91YWE type=del size=4161121 2023-12-03T11:48:39.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.854+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~M9RPPRMNG2JS.0 size=1048576 2023-12-03T11:48:39.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.854+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~M9RPPRMNG2JS.1 size=479354 2023-12-03T11:48:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.856+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=M9RPPRMNG2JS tag=91F0KQOX905G type=write size=2576506 2023-12-03T11:48:39.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.857+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=WPRW7C234WIS type=del size=234923 2023-12-03T11:48:39.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.859+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~94ZYC9MDU2TE.0 size=363785 2023-12-03T11:48:39.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.860+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=94ZYC9MDU2TE tag=Y6QLHE7NHKJO type=write size=1412361 2023-12-03T11:48:39.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.862+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~4JPDISPHRC3N.0 size=777945 2023-12-03T11:48:39.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.862+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=4JPDISPHRC3N tag=ON60J35MX7X8 type=write size=1826521 2023-12-03T11:48:39.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.864+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~1GM36KE3QNNF.0 size=1048576 2023-12-03T11:48:39.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.865+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~1GM36KE3QNNF.1 size=95808 2023-12-03T11:48:39.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.867+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=1GM36KE3QNNF tag=DMB5A1MDIX80 type=write size=2192960 2023-12-03T11:48:39.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.869+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~3V5PGNW07HWH.0 size=1048576 2023-12-03T11:48:39.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.870+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~3V5PGNW07HWH.1 size=408387 2023-12-03T11:48:39.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.870+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=3V5PGNW07HWH tag=060DYP4U5SB0 type=write size=2505539 2023-12-03T11:48:39.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.871+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=V5C10TIX2T63 type=del 2023-12-03T11:48:39.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.873+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~LMXUHAPK1MGX.0 size=154338 2023-12-03T11:48:39.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.873+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=LMXUHAPK1MGX tag=FZSJIL904501 type=write size=1202914 2023-12-03T11:48:39.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.874+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~RDAZHL0D42SE.0 size=1048576 2023-12-03T11:48:39.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.875+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~RDAZHL0D42SE.1 size=405121 2023-12-03T11:48:39.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.877+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=RDAZHL0D42SE tag=71HYLL0435O3 type=write size=2502273 2023-12-03T11:48:39.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.878+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=KJHUO54YUP1D type=del size=702155 2023-12-03T11:48:39.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.880+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=8RP1QVXU4V5I type=del size=4193698 2023-12-03T11:48:39.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.881+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=Z8BNQ1KFLY0U type=del size=375315 2023-12-03T11:48:39.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.883+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~THCX3VWUNGXI.0 size=664158 2023-12-03T11:48:39.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.885+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=THCX3VWUNGXI tag=R69YRCHSMGIM type=write size=1712734 2023-12-03T11:48:39.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.887+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~0ABGKRL7KP9O.0 size=1048576 2023-12-03T11:48:39.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.888+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~0ABGKRL7KP9O.1 size=40215 2023-12-03T11:48:39.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.888+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=0ABGKRL7KP9O tag=IKU2EN0R4E96 type=write size=2137367 2023-12-03T11:48:39.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.889+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=2GR0HO71ZH0E type=write size=391155 2023-12-03T11:48:39.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.891+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=JUE22R752ADU type=del size=4165206 2023-12-03T11:48:39.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.892+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~SRWUITPFOA1S.0 size=60120 2023-12-03T11:48:39.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.894+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=SRWUITPFOA1S tag=3MG6M40VTR5G type=write size=1108696 2023-12-03T11:48:39.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.895+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=FG5S0QQSYMFM type=del size=2932251 2023-12-03T11:48:39.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.896+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=2WK76ANBH9W4 type=del 2023-12-03T11:48:39.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.897+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=NYPNMUG3HNON type=del size=1903051 2023-12-03T11:48:39.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.899+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=YIBE2S00TVBC type=write size=723565 2023-12-03T11:48:39.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.900+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=U61VWV2MLZTB type=write size=535141 2023-12-03T11:48:39.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.901+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.0 size=1048576 2023-12-03T11:48:39.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.902+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.1 size=1048576 2023-12-03T11:48:39.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.903+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~Q3O9L21RD1DO.2 size=825100 2023-12-03T11:48:39.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.906+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=Q3O9L21RD1DO tag=T1RZVCQXAPH7 type=write size=3970828 2023-12-03T11:48:39.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.908+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~KR88S4IGHHVN.0 size=221892 2023-12-03T11:48:39.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.908+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=KR88S4IGHHVN tag=7GO0KCCTSDKC type=write size=1270468 2023-12-03T11:48:39.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.909+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=2VN2D4UVRJYU type=del 2023-12-03T11:48:39.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.912+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.0 size=1048576 2023-12-03T11:48:39.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.913+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.1 size=1048576 2023-12-03T11:48:39.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.914+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.2 size=1025503 2023-12-03T11:48:39.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.914+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=M5DC5H7LJIL3 tag=0Y5FK5C0BXUI type=write size=4171231 2023-12-03T11:48:39.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.916+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~WKAXL5G54VYG.0 size=45913 2023-12-03T11:48:39.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.918+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=WKAXL5G54VYG tag=I2N4GIDGUMNZ type=write size=1094489 2023-12-03T11:48:39.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.919+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=ABIFBICIPZ6Z type=del size=487472 2023-12-03T11:48:39.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.920+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=3UMHM8BZE6F5 type=del 2023-12-03T11:48:39.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.922+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=X7OAN1RZOUN4 type=del size=1442495 2023-12-03T11:48:39.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.923+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=1ZGO2OO91YWE type=del 2023-12-03T11:48:39.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.925+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.0 size=1048576 2023-12-03T11:48:39.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.928+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.1 size=1048576 2023-12-03T11:48:39.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.929+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.2 size=672889 2023-12-03T11:48:39.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.929+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=33XCZU5BKZSD tag=QAOEUR14WKNY type=write size=3818617 2023-12-03T11:48:39.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.930+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=WPRW7C234WIS type=del 2023-12-03T11:48:39.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.931+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=2HPICZ62WZW3 type=del size=1114079 2023-12-03T11:48:39.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.932+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~4JPDISPHRC3N.0 size=777945 2023-12-03T11:48:39.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.934+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=4JPDISPHRC3N tag=ON60J35MX7X8 type=write size=1826521 2023-12-03T11:48:39.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.936+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=28SYE4MEBJWD type=del size=3734229 2023-12-03T11:48:39.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.937+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~3V5PGNW07HWH.0 size=1048576 2023-12-03T11:48:39.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.938+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~3V5PGNW07HWH.1 size=408387 2023-12-03T11:48:39.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.940+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=3V5PGNW07HWH tag=060DYP4U5SB0 type=write size=2505539 2023-12-03T11:48:39.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.941+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=9AZ1DOMT0YJN type=del size=3352727 2023-12-03T11:48:39.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.942+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~LMXUHAPK1MGX.0 size=154338 2023-12-03T11:48:39.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.944+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=LMXUHAPK1MGX tag=FZSJIL904501 type=write size=1202914 2023-12-03T11:48:39.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.946+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~0KRV97SYEF1F.0 size=298474 2023-12-03T11:48:39.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.946+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=0KRV97SYEF1F tag=LXTC2P11OLKG type=write size=1347050 2023-12-03T11:48:39.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.948+0000 7f24574f5780 1 < suggested remove operation key=1V39QA10M1HY tag=KJHUO54YUP1D type=del 2023-12-03T11:48:39.952 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.951+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~I52QMEXOJ7HC.0 size=1048576 2023-12-03T11:48:39.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.952+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~I52QMEXOJ7HC.1 size=293957 2023-12-03T11:48:39.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.952+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=I52QMEXOJ7HC tag=EB2NOCZZRQLZ type=write size=2391109 2023-12-03T11:48:39.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.953+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=Z8BNQ1KFLY0U type=del 2023-12-03T11:48:39.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.956+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.0 size=1048576 2023-12-03T11:48:39.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.957+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.1 size=1048576 2023-12-03T11:48:39.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.958+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.2 size=104775 2023-12-03T11:48:39.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.958+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=8J4M7GA22QUF tag=J6VNZV4FAEY9 type=write size=3250503 2023-12-03T11:48:39.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.959+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~0ABGKRL7KP9O.0 size=1048576 2023-12-03T11:48:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.960+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~0ABGKRL7KP9O.1 size=40215 2023-12-03T11:48:39.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.962+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=0ABGKRL7KP9O tag=IKU2EN0R4E96 type=write size=2137367 2023-12-03T11:48:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.963+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=RAIQ57FDDBSH type=write size=748919 2023-12-03T11:48:39.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.965+0000 7f24574f5780 1 < suggested update operation key=XO42YZR60KQE tag=JUE22R752ADU type=del size=4165206 2023-12-03T11:48:39.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.966+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=YN1UD02ADSGZ type=del size=240043 2023-12-03T11:48:39.969 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.968+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=FG5S0QQSYMFM type=del 2023-12-03T11:48:39.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.969+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=NR6HGWIC2IDK type=write size=631470 2023-12-03T11:48:39.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.970+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=NYPNMUG3HNON type=del 2023-12-03T11:48:39.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.971+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=DKZVX3H7UJKD type=del size=3166193 2023-12-03T11:48:39.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.972+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=U61VWV2MLZTB type=write size=535141: (22) Invalid argument 2023-12-03T11:48:39.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.973+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=9ASCCSG8RU31 type=del size=88991 2023-12-03T11:48:39.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.974+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~KR88S4IGHHVN.0 size=221892 2023-12-03T11:48:39.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.976+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=KR88S4IGHHVN tag=7GO0KCCTSDKC type=write size=1270468 2023-12-03T11:48:39.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.978+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~A4BFE821S6V5.0 size=851917 2023-12-03T11:48:39.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.978+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=A4BFE821S6V5 tag=7KR9E0MJ9ULW type=write size=1900493 2023-12-03T11:48:39.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.979+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.0 size=1048576 2023-12-03T11:48:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.980+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.1 size=1048576 2023-12-03T11:48:39.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.981+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~M5DC5H7LJIL3.2 size=1025503 2023-12-03T11:48:39.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.983+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=M5DC5H7LJIL3 tag=0Y5FK5C0BXUI type=write size=4171231 2023-12-03T11:48:39.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.984+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=FERMSBSN2I9L type=del size=1076698 2023-12-03T11:48:39.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.986+0000 7f24574f5780 1 < canceled operation key=U043YAPGUBT5 tag=ABIFBICIPZ6Z type=del size=487472 2023-12-03T11:48:39.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.987+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=YF4EGP3KTA24 type=del size=3230262 2023-12-03T11:48:39.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.988+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=X7OAN1RZOUN4 type=del 2023-12-03T11:48:39.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.989+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=6FRX2JP3NRQ7 type=del size=3314063 2023-12-03T11:48:39.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.990+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.0 size=1048576 2023-12-03T11:48:39.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.991+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.1 size=1048576 2023-12-03T11:48:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.993+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~33XCZU5BKZSD.2 size=672889 2023-12-03T11:48:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.995+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=33XCZU5BKZSD tag=QAOEUR14WKNY type=write size=3818617 2023-12-03T11:48:39.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.997+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.0 size=1048576 2023-12-03T11:48:39.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.998+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.1 size=1048576 2023-12-03T11:48:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.999+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.2 size=893453 2023-12-03T11:48:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:39.999+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=U07GYQJ48POE tag=PETBHD41Y117 type=write size=4039181 2023-12-03T11:48:40.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.000+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=2HPICZ62WZW3 type=del size=1114079: (22) Invalid argument 2023-12-03T11:48:40.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.002+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.0 size=1048576 2023-12-03T11:48:40.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.003+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.1 size=1048576 2023-12-03T11:48:40.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.004+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.2 size=95123 2023-12-03T11:48:40.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.004+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=1W1J09CLUW4N tag=9VD3W8NTHEZ1 type=write size=3240851 2023-12-03T11:48:40.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.005+0000 7f24574f5780 1 < canceled operation key=MH1SRAATMNXB tag=28SYE4MEBJWD type=del size=3734229 2023-12-03T11:48:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.006+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=628XSKOP9JJQ type=del size=273131 2023-12-03T11:48:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.008+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=9AZ1DOMT0YJN type=del 2023-12-03T11:48:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.009+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=I9DUFIZ1Q680 type=del size=2315066 2023-12-03T11:48:40.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.010+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~0KRV97SYEF1F.0 size=298474 2023-12-03T11:48:40.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.012+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=0KRV97SYEF1F tag=LXTC2P11OLKG type=write size=1347050 2023-12-03T11:48:40.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.013+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=W98V2MGDNU6I type=del size=2118944 2023-12-03T11:48:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.014+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~I52QMEXOJ7HC.0 size=1048576 2023-12-03T11:48:40.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.015+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~I52QMEXOJ7HC.1 size=293957 2023-12-03T11:48:40.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.017+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=I52QMEXOJ7HC tag=EB2NOCZZRQLZ type=write size=2391109 2023-12-03T11:48:40.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.019+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~QSACHCKLIBJY.0 size=1048576 2023-12-03T11:48:40.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.020+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~QSACHCKLIBJY.1 size=901599 2023-12-03T11:48:40.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.020+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=QSACHCKLIBJY tag=VFZSXQ4UTIEX type=write size=2998751 2023-12-03T11:48:40.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.021+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.0 size=1048576 2023-12-03T11:48:40.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.022+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.1 size=1048576 2023-12-03T11:48:40.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.022+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~8J4M7GA22QUF.2 size=104775 2023-12-03T11:48:40.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.025+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=8J4M7GA22QUF tag=J6VNZV4FAEY9 type=write size=3250503 2023-12-03T11:48:40.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.027+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~Z55POKW6U2OF.0 size=1048576 2023-12-03T11:48:40.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.028+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~Z55POKW6U2OF.1 size=596667 2023-12-03T11:48:40.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.028+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=Z55POKW6U2OF tag=1305A34IWZDZ type=write size=2693819 2023-12-03T11:48:40.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.029+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=RAIQ57FDDBSH type=write size=748919 2023-12-03T11:48:40.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.036+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=YNC2SMT0VTRW type=del size=3696907 2023-12-03T11:48:40.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.038+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=YN1UD02ADSGZ type=del 2023-12-03T11:48:40.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.040+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~7O29TSYXLB9Z.0 size=963552 2023-12-03T11:48:40.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.040+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=7O29TSYXLB9Z tag=KCWZRJRHMBDS type=write size=2012128 2023-12-03T11:48:40.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.041+0000 7f24574f5780 1 < canceled operation key=1QR0TE9DXU5Z tag=NR6HGWIC2IDK type=write size=631470 2023-12-03T11:48:40.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.043+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=OLV0X395APJF type=write size=548588 2023-12-03T11:48:40.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.044+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=DKZVX3H7UJKD type=del 2023-12-03T11:48:40.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.045+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=3OL233BT25QW type=del size=4023830 2023-12-03T11:48:40.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.047+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=9ASCCSG8RU31 type=del 2023-12-03T11:48:40.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.049+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.0 size=1048576 2023-12-03T11:48:40.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.050+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.1 size=1048576 2023-12-03T11:48:40.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.051+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.2 size=551326 2023-12-03T11:48:40.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.051+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=MYXP91UR2Z6E tag=VUTKFL121QQJ type=write size=3697054 2023-12-03T11:48:40.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.052+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~A4BFE821S6V5.0 size=851917 2023-12-03T11:48:40.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.054+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=A4BFE821S6V5 tag=7KR9E0MJ9ULW type=write size=1900493 2023-12-03T11:48:40.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.056+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~2XZF18GMF9O7.0 size=751350 2023-12-03T11:48:40.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.056+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=2XZF18GMF9O7 tag=Y93GPT7A8225 type=write size=1799926 2023-12-03T11:48:40.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.057+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=FERMSBSN2I9L type=del 2023-12-03T11:48:40.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.059+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=O6YQ6W6L71MS type=del size=59365 2023-12-03T11:48:40.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.060+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=YF4EGP3KTA24 type=del 2023-12-03T11:48:40.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.062+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.0 size=1048576 2023-12-03T11:48:40.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.063+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.1 size=1048576 2023-12-03T11:48:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.064+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.2 size=18062 2023-12-03T11:48:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.064+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=DZLFK4PNJ66B tag=I287Z09NXK29 type=write size=3163790 2023-12-03T11:48:40.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.065+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=6FRX2JP3NRQ7 type=del 2023-12-03T11:48:40.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.066+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=BK44GZQ7YR7J type=del size=3211624 2023-12-03T11:48:40.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.068+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.0 size=1048576 2023-12-03T11:48:40.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.069+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.1 size=1048576 2023-12-03T11:48:40.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.071+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~U07GYQJ48POE.2 size=893453 2023-12-03T11:48:40.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.074+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=U07GYQJ48POE tag=PETBHD41Y117 type=write size=4039181 2023-12-03T11:48:40.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.075+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=Y06SE8GCAK3F type=del size=710359 2023-12-03T11:48:40.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.076+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.0 size=1048576 2023-12-03T11:48:40.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.077+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.1 size=1048576 2023-12-03T11:48:40.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.077+0000 7f24574f5780 1 < canceled part key=_multipart_XO42YZR60KQE.2~1W1J09CLUW4N.2 size=95123 2023-12-03T11:48:40.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.079+0000 7f24574f5780 1 < canceled multipart upload key=XO42YZR60KQE upload=1W1J09CLUW4N tag=9VD3W8NTHEZ1 type=write size=3240851 2023-12-03T11:48:40.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.080+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=WLSDZFO0STXK type=del size=231814 2023-12-03T11:48:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.082+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=628XSKOP9JJQ type=del size=273131 2023-12-03T11:48:40.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.084+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~5G84EKTOQQBB.0 size=1048576 2023-12-03T11:48:40.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.085+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~5G84EKTOQQBB.1 size=271704 2023-12-03T11:48:40.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.085+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=5G84EKTOQQBB tag=2ZHPNEHIJBXS type=write size=2368856 2023-12-03T11:48:40.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.086+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=I9DUFIZ1Q680 type=del 2023-12-03T11:48:40.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.088+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~MAMW7COF9XC3.0 size=1048576 2023-12-03T11:48:40.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.089+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~MAMW7COF9XC3.1 size=692391 2023-12-03T11:48:40.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.089+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=MAMW7COF9XC3 tag=L7SSYU6E1NUJ type=write size=2789543 2023-12-03T11:48:40.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.091+0000 7f24574f5780 1 < suggested update operation key=1QR0TE9DXU5Z tag=W98V2MGDNU6I type=del size=2118944 2023-12-03T11:48:40.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.092+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=DMQC3HTTRD4Q type=del size=1170557 2023-12-03T11:48:40.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.093+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~QSACHCKLIBJY.0 size=1048576 2023-12-03T11:48:40.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.094+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~QSACHCKLIBJY.1 size=901599 2023-12-03T11:48:40.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.096+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=QSACHCKLIBJY tag=VFZSXQ4UTIEX type=write size=2998751 2023-12-03T11:48:40.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.098+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=1LV2WTN83W1L type=del size=1003257 2023-12-03T11:48:40.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.099+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~Z55POKW6U2OF.0 size=1048576 2023-12-03T11:48:40.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.100+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~Z55POKW6U2OF.1 size=596667 2023-12-03T11:48:40.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.102+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=Z55POKW6U2OF tag=1305A34IWZDZ type=write size=2693819 2023-12-03T11:48:40.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.104+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~MDFLA7ZRWIT3.0 size=173017 2023-12-03T11:48:40.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.104+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=MDFLA7ZRWIT3 tag=K8ALHZTIOQ80 type=write size=1221593 2023-12-03T11:48:40.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.105+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=YNC2SMT0VTRW type=del 2023-12-03T11:48:40.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.107+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~TRVCFCLVYWAX.0 size=1048576 2023-12-03T11:48:40.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.108+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~TRVCFCLVYWAX.1 size=397848 2023-12-03T11:48:40.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.108+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=TRVCFCLVYWAX tag=XTQ754RP22YY type=write size=2495000 2023-12-03T11:48:40.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.109+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~7O29TSYXLB9Z.0 size=963552 2023-12-03T11:48:40.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.111+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=7O29TSYXLB9Z tag=KCWZRJRHMBDS type=write size=2012128 2023-12-03T11:48:40.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.112+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=NBHC64U4U3R2 type=write size=318802 2023-12-03T11:48:40.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.114+0000 7f24574f5780 1 < completed write operation key=35U7YCDYMX46 tag=OLV0X395APJF type=write size=548588 2023-12-03T11:48:40.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.115+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=WRRJB38WFJKY type=del size=528774 2023-12-03T11:48:40.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.116+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=3OL233BT25QW type=del 2023-12-03T11:48:40.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.117+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=RDAPZG5QQGFU type=write size=963078 2023-12-03T11:48:40.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.119+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.0 size=1048576 2023-12-03T11:48:40.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.119+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.1 size=1048576 2023-12-03T11:48:40.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.120+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MYXP91UR2Z6E.2 size=551326 2023-12-03T11:48:40.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.122+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=MYXP91UR2Z6E tag=VUTKFL121QQJ type=write size=3697054 2023-12-03T11:48:40.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.124+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=97DOULWKBQ4C type=write size=558581 2023-12-03T11:48:40.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.125+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~2XZF18GMF9O7.0 size=751350 2023-12-03T11:48:40.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.127+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=2XZF18GMF9O7 tag=Y93GPT7A8225 type=write size=1799926 2023-12-03T11:48:40.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.129+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~I9U76EU5WW61.0 size=1048576 2023-12-03T11:48:40.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.130+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~I9U76EU5WW61.1 size=8887 2023-12-03T11:48:40.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.130+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=I9U76EU5WW61 tag=RC645BGX9VI5 type=write size=2106039 2023-12-03T11:48:40.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.132+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=O6YQ6W6L71MS type=del 2023-12-03T11:48:40.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.133+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=O14WPB8SRWHZ type=del size=1017214 2023-12-03T11:48:40.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.135+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.0 size=1048576 2023-12-03T11:48:40.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.136+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.1 size=1048576 2023-12-03T11:48:40.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.137+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~DZLFK4PNJ66B.2 size=18062 2023-12-03T11:48:40.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.139+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=DZLFK4PNJ66B tag=I287Z09NXK29 type=write size=3163790 2023-12-03T11:48:40.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.141+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.0 size=1048576 2023-12-03T11:48:40.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.142+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.1 size=1048576 2023-12-03T11:48:40.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.143+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.2 size=923437 2023-12-03T11:48:40.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.143+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=AJB7MOI2I5S1 tag=B1PA5A7SMDNS type=write size=4069165 2023-12-03T11:48:40.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.144+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=BK44GZQ7YR7J type=del 2023-12-03T11:48:40.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.145+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=ULPWZO4BGATR type=write size=67247 2023-12-03T11:48:40.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.146+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=Y06SE8GCAK3F type=del 2023-12-03T11:48:40.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.147+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=NACQM7ALFT7P type=del size=169401 2023-12-03T11:48:40.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.149+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=WLSDZFO0STXK type=del 2023-12-03T11:48:40.151 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.150+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=RBVKCQ55WKXW type=del size=1939019 2023-12-03T11:48:40.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.151+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~5G84EKTOQQBB.0 size=1048576 2023-12-03T11:48:40.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.152+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~5G84EKTOQQBB.1 size=271704 2023-12-03T11:48:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.154+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=5G84EKTOQQBB tag=2ZHPNEHIJBXS type=write size=2368856 2023-12-03T11:48:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.156+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.0 size=1048576 2023-12-03T11:48:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.157+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.1 size=1048576 2023-12-03T11:48:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.158+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.2 size=57518 2023-12-03T11:48:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.158+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=3R3L1K7WV1LG tag=GGG6VZY5YLWO type=write size=3203246 2023-12-03T11:48:40.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.159+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~MAMW7COF9XC3.0 size=1048576 2023-12-03T11:48:40.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.160+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~MAMW7COF9XC3.1 size=692391 2023-12-03T11:48:40.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.162+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=MAMW7COF9XC3 tag=L7SSYU6E1NUJ type=write size=2789543 2023-12-03T11:48:40.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.163+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=A8604UB3KF7E type=write size=665877 2023-12-03T11:48:40.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.164+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=DMQC3HTTRD4Q type=del size=1170557 2023-12-03T11:48:40.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.165+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=J1TP0A2MK4Q9 type=del size=1403354 2023-12-03T11:48:40.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.166+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=1LV2WTN83W1L type=del 2023-12-03T11:48:40.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.168+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~3OIL7304BO9V.0 size=764277 2023-12-03T11:48:40.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.168+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=3OIL7304BO9V tag=C7KHDIB3PWNO type=write size=1812853 2023-12-03T11:48:40.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.169+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~MDFLA7ZRWIT3.0 size=173017 2023-12-03T11:48:40.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.170+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=MDFLA7ZRWIT3 tag=K8ALHZTIOQ80 type=write size=1221593 2023-12-03T11:48:40.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.172+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=W98VXVE1NPL1 type=del size=2568605 2023-12-03T11:48:40.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.173+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~TRVCFCLVYWAX.0 size=1048576 2023-12-03T11:48:40.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.174+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~TRVCFCLVYWAX.1 size=397848 2023-12-03T11:48:40.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.175+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=TRVCFCLVYWAX tag=XTQ754RP22YY type=write size=2495000 2023-12-03T11:48:40.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.177+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=07L0PTCS7KJZ type=del size=1238674 2023-12-03T11:48:40.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.178+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=NBHC64U4U3R2 type=write size=318802 2023-12-03T11:48:40.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.180+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=ZUVNG2234CHX type=del size=276676 2023-12-03T11:48:40.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.181+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=WRRJB38WFJKY type=del 2023-12-03T11:48:40.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.183+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~SELGTTJ16ESE.0 size=761543 2023-12-03T11:48:40.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.183+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=SELGTTJ16ESE tag=O419KI2T3T50 type=write size=1810119 2023-12-03T11:48:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.185+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=RDAPZG5QQGFU type=write size=963078 2023-12-03T11:48:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.187+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~8WMRDAP5N0L9.0 size=1048576 2023-12-03T11:48:40.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.188+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~8WMRDAP5N0L9.1 size=56066 2023-12-03T11:48:40.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.188+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=8WMRDAP5N0L9 tag=QWIKYMN8WYX2 type=write size=2153218 2023-12-03T11:48:40.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.189+0000 7f24574f5780 -1 < failed to complete operation key=RABUR02D2ZMD tag=97DOULWKBQ4C type=write size=558581: (22) Invalid argument 2023-12-03T11:48:40.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.190+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=L9V6T81YESD0 type=del size=89175 2023-12-03T11:48:40.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.191+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~I9U76EU5WW61.0 size=1048576 2023-12-03T11:48:40.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.192+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~I9U76EU5WW61.1 size=8887 2023-12-03T11:48:40.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.194+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=I9U76EU5WW61 tag=RC645BGX9VI5 type=write size=2106039 2023-12-03T11:48:40.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.198+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.0 size=1048576 2023-12-03T11:48:40.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.198+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.1 size=1048576 2023-12-03T11:48:40.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.199+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.2 size=633059 2023-12-03T11:48:40.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.199+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=9HCVYK16CTQQ tag=13O91BOBJB7X type=write size=3778787 2023-12-03T11:48:40.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.200+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=O14WPB8SRWHZ type=del 2023-12-03T11:48:40.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.202+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=9ORUNTRW0U8A type=del size=3736440 2023-12-03T11:48:40.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.202+0000 7f24574f5780 1 < canceled part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.0 size=1048576 2023-12-03T11:48:40.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.202+0000 7f24574f5780 1 < canceled part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.1 size=1048576 2023-12-03T11:48:40.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.203+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~AJB7MOI2I5S1.2 size=923437 2023-12-03T11:48:40.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.205+0000 7f24574f5780 1 < canceled multipart upload key=S0EH0Q0UA8HG upload=AJB7MOI2I5S1 tag=B1PA5A7SMDNS type=write size=4069165 2023-12-03T11:48:40.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.207+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~A5LMVF0TSF7I.0 size=1048576 2023-12-03T11:48:40.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.208+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~A5LMVF0TSF7I.1 size=1020566 2023-12-03T11:48:40.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.208+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=A5LMVF0TSF7I tag=X2IO5Q8AI5D7 type=write size=3117718 2023-12-03T11:48:40.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.209+0000 7f24574f5780 1 < completed write operation key=863EOZDYRWH3 tag=ULPWZO4BGATR type=write size=67247 2023-12-03T11:48:40.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.210+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=5MKDHSVBMOY9 type=del size=3136576 2023-12-03T11:48:40.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.212+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=NACQM7ALFT7P type=del size=169401 2023-12-03T11:48:40.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.213+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=5ZO1GLMOP196 type=del size=1033699 2023-12-03T11:48:40.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.214+0000 7f24574f5780 1 < canceled operation key=MH1SRAATMNXB tag=RBVKCQ55WKXW type=del size=1939019 2023-12-03T11:48:40.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.215+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=6GBN45OZ5J8F type=del size=3981638 2023-12-03T11:48:40.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.217+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.0 size=1048576 2023-12-03T11:48:40.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.218+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.1 size=1048576 2023-12-03T11:48:40.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.219+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~3R3L1K7WV1LG.2 size=57518 2023-12-03T11:48:40.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.221+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=3R3L1K7WV1LG tag=GGG6VZY5YLWO type=write size=3203246 2023-12-03T11:48:40.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.222+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=KBJGRUNC80UJ type=del size=1835702 2023-12-03T11:48:40.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.223+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=A8604UB3KF7E type=write size=665877 2023-12-03T11:48:40.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.224+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=RX566EHTUQRR type=del size=322559 2023-12-03T11:48:40.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.225+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=J1TP0A2MK4Q9 type=del 2023-12-03T11:48:40.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.226+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=YS0AT0N3I7LT type=del size=924206 2023-12-03T11:48:40.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.227+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~3OIL7304BO9V.0 size=764277 2023-12-03T11:48:40.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.229+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=3OIL7304BO9V tag=C7KHDIB3PWNO type=write size=1812853 2023-12-03T11:48:40.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.231+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~S14ICKGVWFPC.0 size=1048576 2023-12-03T11:48:40.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.232+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~S14ICKGVWFPC.1 size=536749 2023-12-03T11:48:40.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.232+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=S14ICKGVWFPC tag=XEKTRWK4CY69 type=write size=2633901 2023-12-03T11:48:40.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.233+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=W98VXVE1NPL1 type=del 2023-12-03T11:48:40.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.235+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=86OLJTH2UZGX type=del size=3975778 2023-12-03T11:48:40.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.237+0000 7f24574f5780 1 < suggested remove operation key=1V1C59DP8YPF tag=07L0PTCS7KJZ type=del 2023-12-03T11:48:40.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.239+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.0 size=1048576 2023-12-03T11:48:40.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.240+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.1 size=1048576 2023-12-03T11:48:40.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.241+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.2 size=1011337 2023-12-03T11:48:40.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.241+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=UELAZ1M9JN76 tag=DUH8RWWJ1PO7 type=write size=4157065 2023-12-03T11:48:40.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.244+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=ZUVNG2234CHX type=del 2023-12-03T11:48:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.245+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=IY0BWPWAR4WC type=write size=468108 2023-12-03T11:48:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.245+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~SELGTTJ16ESE.0 size=761543 2023-12-03T11:48:40.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.247+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=SELGTTJ16ESE tag=O419KI2T3T50 type=write size=1810119 2023-12-03T11:48:40.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.248+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=EY43GV2315DO type=del size=234944 2023-12-03T11:48:40.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.250+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~8WMRDAP5N0L9.0 size=1048576 2023-12-03T11:48:40.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.251+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~8WMRDAP5N0L9.1 size=56066 2023-12-03T11:48:40.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.253+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=8WMRDAP5N0L9 tag=QWIKYMN8WYX2 type=write size=2153218 2023-12-03T11:48:40.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.254+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=22DRFYVNEGRR type=del size=569754 2023-12-03T11:48:40.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.255+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=L9V6T81YESD0 type=del 2023-12-03T11:48:40.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.257+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~SW2QSHQ42E94.0 size=282403 2023-12-03T11:48:40.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.257+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=SW2QSHQ42E94 tag=5IM894KYANLN type=write size=1330979 2023-12-03T11:48:40.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.258+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.0 size=1048576 2023-12-03T11:48:40.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.259+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.1 size=1048576 2023-12-03T11:48:40.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.262+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~9HCVYK16CTQQ.2 size=633059 2023-12-03T11:48:40.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.264+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=9HCVYK16CTQQ tag=13O91BOBJB7X type=write size=3778787 2023-12-03T11:48:40.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.266+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~C1IYEJPTDCEV.0 size=1048576 2023-12-03T11:48:40.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.267+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~C1IYEJPTDCEV.1 size=930190 2023-12-03T11:48:40.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.267+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=C1IYEJPTDCEV tag=OXC9XE1ENV51 type=write size=3027342 2023-12-03T11:48:40.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.270+0000 7f24574f5780 1 < suggested update operation key=DA6ZXCLKTC1L tag=9ORUNTRW0U8A type=del size=3736440 2023-12-03T11:48:40.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.272+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.0 size=1048576 2023-12-03T11:48:40.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.273+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.1 size=1048576 2023-12-03T11:48:40.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.274+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.2 size=139117 2023-12-03T11:48:40.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.274+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=MI69POVF1RXL tag=ETMJ4J33DR7V type=write size=3284845 2023-12-03T11:48:40.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.275+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~A5LMVF0TSF7I.0 size=1048576 2023-12-03T11:48:40.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.276+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~A5LMVF0TSF7I.1 size=1020566 2023-12-03T11:48:40.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.278+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=A5LMVF0TSF7I tag=X2IO5Q8AI5D7 type=write size=3117718 2023-12-03T11:48:40.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.280+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~J4N8WWV903S9.0 size=1048576 2023-12-03T11:48:40.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.281+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~J4N8WWV903S9.1 size=214781 2023-12-03T11:48:40.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.281+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=J4N8WWV903S9 tag=L427EMARPG48 type=write size=2311933 2023-12-03T11:48:40.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.282+0000 7f24574f5780 1 < canceled operation key=863EOZDYRWH3 tag=5MKDHSVBMOY9 type=del size=3136576 2023-12-03T11:48:40.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.283+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=QAXN422PKBSL type=del size=2478546 2023-12-03T11:48:40.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.284+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=5ZO1GLMOP196 type=del 2023-12-03T11:48:40.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.285+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=OIECB6WROXTG type=del size=3351620 2023-12-03T11:48:40.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.287+0000 7f24574f5780 1 < canceled operation key=ON2E6H35N9FJ tag=6GBN45OZ5J8F type=del size=3981638 2023-12-03T11:48:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.288+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=POXSKYSR6CCJ type=write size=195058 2023-12-03T11:48:40.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.289+0000 7f24574f5780 1 < canceled operation key=BV8C0VSJ1GTW tag=KBJGRUNC80UJ type=del size=1835702 2023-12-03T11:48:40.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.291+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=QKO36Y7ZEQGL type=del size=1341312 2023-12-03T11:48:40.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.295+0000 7f24574f5780 1 < suggested remove operation key=863EOZDYRWH3 tag=RX566EHTUQRR type=del 2023-12-03T11:48:40.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.297+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~ZL6ZNCQM2YU8.0 size=174077 2023-12-03T11:48:40.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.297+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=ZL6ZNCQM2YU8 tag=BVM6K2VD89CU type=write size=1222653 2023-12-03T11:48:40.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.298+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=YS0AT0N3I7LT type=del 2023-12-03T11:48:40.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.299+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=YCL13B5QKHTK type=del size=3826682 2023-12-03T11:48:40.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.300+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~S14ICKGVWFPC.0 size=1048576 2023-12-03T11:48:40.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.301+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~S14ICKGVWFPC.1 size=536749 2023-12-03T11:48:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.303+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=S14ICKGVWFPC tag=XEKTRWK4CY69 type=write size=2633901 2023-12-03T11:48:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.304+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=84NI5U9CKIFN type=del size=1955792 2023-12-03T11:48:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.305+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=86OLJTH2UZGX type=del 2023-12-03T11:48:40.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.307+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.0 size=1048576 2023-12-03T11:48:40.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.308+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.1 size=1048576 2023-12-03T11:48:40.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.309+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.2 size=98535 2023-12-03T11:48:40.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.309+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=F56RS71K7H0U tag=8U22V33DFX5A type=write size=3244263 2023-12-03T11:48:40.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.310+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.0 size=1048576 2023-12-03T11:48:40.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.311+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.1 size=1048576 2023-12-03T11:48:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.312+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~UELAZ1M9JN76.2 size=1011337 2023-12-03T11:48:40.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.314+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=UELAZ1M9JN76 tag=DUH8RWWJ1PO7 type=write size=4157065 2023-12-03T11:48:40.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.315+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=YACH16P6RV3J type=write size=335263 2023-12-03T11:48:40.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.317+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=IY0BWPWAR4WC type=write size=468108 2023-12-03T11:48:40.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.318+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=APPWWDG7UEU0 type=del size=1284023 2023-12-03T11:48:40.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.320+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=EY43GV2315DO type=del size=234944 2023-12-03T11:48:40.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.322+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~SLAMYJROO2KC.0 size=861551 2023-12-03T11:48:40.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.322+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=SLAMYJROO2KC tag=87RUOGG3REMA type=write size=1910127 2023-12-03T11:48:40.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.325+0000 7f24574f5780 1 < suggested update operation key=6WY91DFIRUN8 tag=22DRFYVNEGRR type=del size=569754 2023-12-03T11:48:40.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.326+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=DJYEY8B8FURL type=del size=3587738 2023-12-03T11:48:40.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.327+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~SW2QSHQ42E94.0 size=282403 2023-12-03T11:48:40.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.329+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=SW2QSHQ42E94 tag=5IM894KYANLN type=write size=1330979 2023-12-03T11:48:40.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.330+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=EDDGC8W45X57 type=write size=937373 2023-12-03T11:48:40.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.331+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~C1IYEJPTDCEV.0 size=1048576 2023-12-03T11:48:40.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.332+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~C1IYEJPTDCEV.1 size=930190 2023-12-03T11:48:40.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.334+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=C1IYEJPTDCEV tag=OXC9XE1ENV51 type=write size=3027342 2023-12-03T11:48:40.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.336+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~UWTRWSZ61LEY.0 size=752237 2023-12-03T11:48:40.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.336+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=UWTRWSZ61LEY tag=2MIWZFK98IRJ type=write size=1800813 2023-12-03T11:48:40.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.337+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.0 size=1048576 2023-12-03T11:48:40.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.338+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.1 size=1048576 2023-12-03T11:48:40.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.339+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~MI69POVF1RXL.2 size=139117 2023-12-03T11:48:40.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.341+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=MI69POVF1RXL tag=ETMJ4J33DR7V type=write size=3284845 2023-12-03T11:48:40.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.343+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~3UNDTJXYUPEO.0 size=918655 2023-12-03T11:48:40.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.343+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=3UNDTJXYUPEO tag=V9KKX76ZNPIE type=write size=1967231 2023-12-03T11:48:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.344+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~J4N8WWV903S9.0 size=1048576 2023-12-03T11:48:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.344+0000 7f24574f5780 1 < canceled part key=_multipart_A4LOQE7VRDN7.2~J4N8WWV903S9.1 size=214781 2023-12-03T11:48:40.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.346+0000 7f24574f5780 1 < canceled multipart upload key=A4LOQE7VRDN7 upload=J4N8WWV903S9 tag=L427EMARPG48 type=write size=2311933 2023-12-03T11:48:40.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.347+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=DHFWZ9UTAWBJ type=del size=2043379 2023-12-03T11:48:40.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.349+0000 7f24574f5780 1 < canceled operation key=WZMWCSAZX8RB tag=QAXN422PKBSL type=del size=2478546 2023-12-03T11:48:40.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.351+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~91E8Y6BMG63E.0 size=1048576 2023-12-03T11:48:40.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.352+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~91E8Y6BMG63E.1 size=860030 2023-12-03T11:48:40.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.352+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=91E8Y6BMG63E tag=WBC2GY1L5ESX type=write size=2957182 2023-12-03T11:48:40.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.353+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=OIECB6WROXTG type=del 2023-12-03T11:48:40.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.354+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=3LWZ6NBOO0Y7 type=del size=1226314 2023-12-03T11:48:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.355+0000 7f24574f5780 1 < completed write operation key=ON2E6H35N9FJ tag=POXSKYSR6CCJ type=write size=195058 2023-12-03T11:48:40.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.357+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=FIHD7SOA1A9U type=del size=3786614 2023-12-03T11:48:40.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.358+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=QKO36Y7ZEQGL type=del 2023-12-03T11:48:40.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.359+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=SO2305TPLTI7 type=del size=2159090 2023-12-03T11:48:40.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.360+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~ZL6ZNCQM2YU8.0 size=174077 2023-12-03T11:48:40.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.362+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=ZL6ZNCQM2YU8 tag=BVM6K2VD89CU type=write size=1222653 2023-12-03T11:48:40.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.363+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=YFKW6F4LVNIH type=del size=3783604 2023-12-03T11:48:40.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.364+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=YCL13B5QKHTK type=del 2023-12-03T11:48:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.365+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=WY9PF6IMVZLK type=del size=64535 2023-12-03T11:48:40.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.366+0000 7f24574f5780 1 < canceled operation key=U043YAPGUBT5 tag=84NI5U9CKIFN type=del size=1955792 2023-12-03T11:48:40.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.368+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=2ZHKP52FJQOP type=del size=72929 2023-12-03T11:48:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.369+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.0 size=1048576 2023-12-03T11:48:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.370+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.1 size=1048576 2023-12-03T11:48:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.371+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~F56RS71K7H0U.2 size=98535 2023-12-03T11:48:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.373+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=F56RS71K7H0U tag=8U22V33DFX5A type=write size=3244263 2023-12-03T11:48:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.374+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=YJOK71T2V7U8 type=del size=2970704 2023-12-03T11:48:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.375+0000 7f24574f5780 -1 < failed to complete operation key=S7Q5TCM499H1 tag=YACH16P6RV3J type=write size=335263: (22) Invalid argument 2023-12-03T11:48:40.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.376+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=4KI2B76Y6NQK type=write size=645119 2023-12-03T11:48:40.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.377+0000 7f24574f5780 1 < canceled operation key=PGTGVLK4DBN0 tag=APPWWDG7UEU0 type=del size=1284023 2023-12-03T11:48:40.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.380+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.0 size=1048576 2023-12-03T11:48:40.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.381+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.1 size=1048576 2023-12-03T11:48:40.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.382+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.2 size=647850 2023-12-03T11:48:40.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.382+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=ZUYIRW20JPGH tag=VPJY2PBUMGSI type=write size=3793578 2023-12-03T11:48:40.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.383+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~SLAMYJROO2KC.0 size=861551 2023-12-03T11:48:40.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.385+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=SLAMYJROO2KC tag=87RUOGG3REMA type=write size=1910127 2023-12-03T11:48:40.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.387+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~GGYTVXG91D1Q.0 size=1048576 2023-12-03T11:48:40.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.388+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~GGYTVXG91D1Q.1 size=144551 2023-12-03T11:48:40.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.388+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=GGYTVXG91D1Q tag=YRZ92IUOD5XZ type=write size=2241703 2023-12-03T11:48:40.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.390+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=DJYEY8B8FURL type=del size=3587738 2023-12-03T11:48:40.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.392+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=2FCXGH67411S type=write size=642605 2023-12-03T11:48:40.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.393+0000 7f24574f5780 1 < completed write operation key=863EOZDYRWH3 tag=EDDGC8W45X57 type=write size=937373 2023-12-03T11:48:40.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.394+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=T2BYQWO4OR1C type=del size=2350057 2023-12-03T11:48:40.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.395+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~UWTRWSZ61LEY.0 size=752237 2023-12-03T11:48:40.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.397+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=UWTRWSZ61LEY tag=2MIWZFK98IRJ type=write size=1800813 2023-12-03T11:48:40.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.399+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~OV3SMQ1B2VIR.0 size=211236 2023-12-03T11:48:40.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.399+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=OV3SMQ1B2VIR tag=Q3K5YAYUXIC3 type=write size=1259812 2023-12-03T11:48:40.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.402+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~3UNDTJXYUPEO.0 size=918655 2023-12-03T11:48:40.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.405+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=3UNDTJXYUPEO tag=V9KKX76ZNPIE type=write size=1967231 2023-12-03T11:48:40.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.407+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.0 size=1048576 2023-12-03T11:48:40.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.408+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.1 size=1048576 2023-12-03T11:48:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.409+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.2 size=110091 2023-12-03T11:48:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.409+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=J97C0JBP9WU1 tag=PWPD4VT66QB5 type=write size=3255819 2023-12-03T11:48:40.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.410+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=DHFWZ9UTAWBJ type=del 2023-12-03T11:48:40.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.411+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=X3TXINWIC1CE type=del size=3945375 2023-12-03T11:48:40.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.413+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~91E8Y6BMG63E.0 size=1048576 2023-12-03T11:48:40.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.414+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~91E8Y6BMG63E.1 size=860030 2023-12-03T11:48:40.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.416+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=91E8Y6BMG63E tag=WBC2GY1L5ESX type=write size=2957182 2023-12-03T11:48:40.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.417+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=7KH4NNX6HR8H type=write size=218980 2023-12-03T11:48:40.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.418+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=3LWZ6NBOO0Y7 type=del 2023-12-03T11:48:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.420+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~0Y027IIJ96X0.0 size=1048576 2023-12-03T11:48:40.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.421+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~0Y027IIJ96X0.1 size=165772 2023-12-03T11:48:40.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.421+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=0Y027IIJ96X0 tag=TTLUN2BRJQ00 type=write size=2262924 2023-12-03T11:48:40.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.423+0000 7f24574f5780 1 < canceled operation key=PGTGVLK4DBN0 tag=FIHD7SOA1A9U type=del size=3786614 2023-12-03T11:48:40.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.425+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~S5S7U0Z69TYA.0 size=231806 2023-12-03T11:48:40.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.425+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=S5S7U0Z69TYA tag=P23P36M6X3UL type=write size=1280382 2023-12-03T11:48:40.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.426+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=SO2305TPLTI7 type=del 2023-12-03T11:48:40.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.427+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=9EKYUO0L8SFX type=del size=3967032 2023-12-03T11:48:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.429+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=YFKW6F4LVNIH type=del 2023-12-03T11:48:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.431+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=7V3H4QRTL2XY type=del size=3412399 2023-12-03T11:48:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.433+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=WY9PF6IMVZLK type=del size=64535 2023-12-03T11:48:40.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.434+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=292LC5P6208F type=del size=2630801 2023-12-03T11:48:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.437+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=2ZHKP52FJQOP type=del 2023-12-03T11:48:40.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.439+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.0 size=1048576 2023-12-03T11:48:40.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.440+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.1 size=1048576 2023-12-03T11:48:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.441+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.2 size=804072 2023-12-03T11:48:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.441+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=1UZWJSL09HKK tag=7KPAI76IBYUT type=write size=3949800 2023-12-03T11:48:40.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.443+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=YJOK71T2V7U8 type=del 2023-12-03T11:48:40.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.444+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=3P9WHWCI04HS type=del size=4013778 2023-12-03T11:48:40.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.446+0000 7f24574f5780 1 < suggested remove operation key=MY3K9M4QRS3L tag=4KI2B76Y6NQK type=write 2023-12-03T11:48:40.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.448+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=STWSM3EZVPQE type=del size=2288660 2023-12-03T11:48:40.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.449+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.0 size=1048576 2023-12-03T11:48:40.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.450+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.1 size=1048576 2023-12-03T11:48:40.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.451+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~ZUYIRW20JPGH.2 size=647850 2023-12-03T11:48:40.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.453+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=ZUYIRW20JPGH tag=VPJY2PBUMGSI type=write size=3793578 2023-12-03T11:48:40.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.455+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~GWT4FC3C75B5.0 size=1048576 2023-12-03T11:48:40.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.456+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~GWT4FC3C75B5.1 size=836636 2023-12-03T11:48:40.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.456+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=GWT4FC3C75B5 tag=UPZNFEF27ASJ type=write size=2933788 2023-12-03T11:48:40.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.456+0000 7f24574f5780 1 < canceled part key=_multipart_DA6ZXCLKTC1L.2~GGYTVXG91D1Q.0 size=1048576 2023-12-03T11:48:40.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.458+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~GGYTVXG91D1Q.1 size=144551 2023-12-03T11:48:40.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.460+0000 7f24574f5780 1 < canceled multipart upload key=DA6ZXCLKTC1L upload=GGYTVXG91D1Q tag=YRZ92IUOD5XZ type=write size=2241703 2023-12-03T11:48:40.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.461+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=JLMQT06S7IWB type=write size=329116 2023-12-03T11:48:40.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.464+0000 7f24574f5780 1 < suggested remove operation key=MH1SRAATMNXB tag=2FCXGH67411S type=write 2023-12-03T11:48:40.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.466+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~PX12BK5DDUB3.0 size=152710 2023-12-03T11:48:40.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.466+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=PX12BK5DDUB3 tag=AAYR0RDRVVR3 type=write size=1201286 2023-12-03T11:48:40.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.466+0000 7f24574f5780 -1 < no bi entry to suggest for operation key=MH1SRAATMNXB tag=T2BYQWO4OR1C type=del size=2350057: (2) No such file or directory 2023-12-03T11:48:40.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.468+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~V0U2BXSIETOV.0 size=1048576 2023-12-03T11:48:40.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.469+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~V0U2BXSIETOV.1 size=245082 2023-12-03T11:48:40.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.469+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=V0U2BXSIETOV tag=0B5REHXT5908 type=write size=2342234 2023-12-03T11:48:40.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.471+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~OV3SMQ1B2VIR.0 size=211236 2023-12-03T11:48:40.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.473+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=OV3SMQ1B2VIR tag=Q3K5YAYUXIC3 type=write size=1259812 2023-12-03T11:48:40.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.475+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~P23AJ5XA5T0C.0 size=1048576 2023-12-03T11:48:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.475+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~P23AJ5XA5T0C.1 size=677556 2023-12-03T11:48:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.475+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=P23AJ5XA5T0C tag=6RREPMW159KK type=write size=2774708 2023-12-03T11:48:40.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.477+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.0 size=1048576 2023-12-03T11:48:40.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.478+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.1 size=1048576 2023-12-03T11:48:40.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.479+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~J97C0JBP9WU1.2 size=110091 2023-12-03T11:48:40.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.481+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=J97C0JBP9WU1 tag=PWPD4VT66QB5 type=write size=3255819 2023-12-03T11:48:40.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.482+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=DNMQWFLDTKRW type=del size=680140 2023-12-03T11:48:40.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.483+0000 7f24574f5780 1 < canceled operation key=PS1OYWI44L80 tag=X3TXINWIC1CE type=del size=3945375 2023-12-03T11:48:40.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.484+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=E0EA77I61BZB type=write size=286140 2023-12-03T11:48:40.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.485+0000 7f24574f5780 1 < completed write operation key=9LKFY1U0JRBQ tag=7KH4NNX6HR8H type=write size=218980 2023-12-03T11:48:40.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.487+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~0BZ7KCL4NNEN.0 size=1048576 2023-12-03T11:48:40.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.488+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~0BZ7KCL4NNEN.1 size=248155 2023-12-03T11:48:40.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.488+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=0BZ7KCL4NNEN tag=ZX8DFJ3J296O type=write size=2345307 2023-12-03T11:48:40.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.490+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~0Y027IIJ96X0.0 size=1048576 2023-12-03T11:48:40.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.491+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~0Y027IIJ96X0.1 size=165772 2023-12-03T11:48:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.492+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=0Y027IIJ96X0 tag=TTLUN2BRJQ00 type=write size=2262924 2023-12-03T11:48:40.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.494+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=S6SNG6MJGJS6 type=del size=4105109 2023-12-03T11:48:40.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.495+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~S5S7U0Z69TYA.0 size=231806 2023-12-03T11:48:40.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.497+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=S5S7U0Z69TYA tag=P23P36M6X3UL type=write size=1280382 2023-12-03T11:48:40.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.498+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=FHUN4105T37W type=del size=1222211 2023-12-03T11:48:40.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.499+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=9EKYUO0L8SFX type=del 2023-12-03T11:48:40.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.502+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~2DMTWVFRFM2S.0 size=1048576 2023-12-03T11:48:40.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.503+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~2DMTWVFRFM2S.1 size=256764 2023-12-03T11:48:40.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.503+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=2DMTWVFRFM2S tag=OTX4Q481M0K7 type=write size=2353916 2023-12-03T11:48:40.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.504+0000 7f24574f5780 -1 < failed to complete operation key=MH1SRAATMNXB tag=7V3H4QRTL2XY type=del size=3412399: (22) Invalid argument 2023-12-03T11:48:40.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.505+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=OJI67QIO1F82 type=write size=819478 2023-12-03T11:48:40.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.506+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=292LC5P6208F type=del 2023-12-03T11:48:40.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.508+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=5GT32LHXKP10 type=del size=3623 2023-12-03T11:48:40.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.509+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.0 size=1048576 2023-12-03T11:48:40.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.510+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.1 size=1048576 2023-12-03T11:48:40.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.511+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~1UZWJSL09HKK.2 size=804072 2023-12-03T11:48:40.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.513+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=1UZWJSL09HKK tag=7KPAI76IBYUT type=write size=3949800 2023-12-03T11:48:40.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.515+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=OYWVN6FTA8O7 type=write size=690568 2023-12-03T11:48:40.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.517+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=3P9WHWCI04HS type=del 2023-12-03T11:48:40.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.519+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=FIE5TEC5RZHT type=del size=2672153 2023-12-03T11:48:40.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.520+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=STWSM3EZVPQE type=del 2023-12-03T11:48:40.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.525+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.0 size=1048576 2023-12-03T11:48:40.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.526+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.1 size=1048576 2023-12-03T11:48:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.527+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.2 size=765815 2023-12-03T11:48:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.527+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=GXG3MJRVTD67 tag=ZDP9MB33CN9H type=write size=3911543 2023-12-03T11:48:40.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.528+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~GWT4FC3C75B5.0 size=1048576 2023-12-03T11:48:40.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.529+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~GWT4FC3C75B5.1 size=836636 2023-12-03T11:48:40.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.531+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=GWT4FC3C75B5 tag=UPZNFEF27ASJ type=write size=2933788 2023-12-03T11:48:40.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.533+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~SK0M6P6FX5N8.0 size=1048576 2023-12-03T11:48:40.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.534+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~SK0M6P6FX5N8.1 size=736186 2023-12-03T11:48:40.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.534+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=SK0M6P6FX5N8 tag=COJM1RKRX8E0 type=write size=2833338 2023-12-03T11:48:40.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.536+0000 7f24574f5780 1 < suggested remove operation key=SC8UOVLQQ0VD tag=JLMQT06S7IWB type=write 2023-12-03T11:48:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.538+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~0G5924C4DFNX.0 size=257895 2023-12-03T11:48:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.538+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=0G5924C4DFNX tag=P3ZOXPX3ORGE type=write size=1306471 2023-12-03T11:48:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.539+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~PX12BK5DDUB3.0 size=152710 2023-12-03T11:48:40.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.541+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=PX12BK5DDUB3 tag=AAYR0RDRVVR3 type=write size=1201286 2023-12-03T11:48:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.543+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=4YBCFHD4GEU4 type=del size=3495225 2023-12-03T11:48:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.544+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~V0U2BXSIETOV.0 size=1048576 2023-12-03T11:48:40.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.545+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~V0U2BXSIETOV.1 size=245082 2023-12-03T11:48:40.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.547+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=V0U2BXSIETOV tag=0B5REHXT5908 type=write size=2342234 2023-12-03T11:48:40.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.548+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=MPT2LYPNG46S type=del size=4121615 2023-12-03T11:48:40.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.549+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~P23AJ5XA5T0C.0 size=1048576 2023-12-03T11:48:40.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.550+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~P23AJ5XA5T0C.1 size=677556 2023-12-03T11:48:40.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.552+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=P23AJ5XA5T0C tag=6RREPMW159KK type=write size=2774708 2023-12-03T11:48:40.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.553+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=CR7XI6WIYZ6P type=del size=3540822 2023-12-03T11:48:40.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.555+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=DNMQWFLDTKRW type=del 2023-12-03T11:48:40.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.556+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=745SW53HNQG0 type=del size=4035747 2023-12-03T11:48:40.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.557+0000 7f24574f5780 1 < canceled operation key=PGTGVLK4DBN0 tag=E0EA77I61BZB type=write size=286140 2023-12-03T11:48:40.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.558+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=GH009D2WFCAC type=del size=2627326 2023-12-03T11:48:40.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.559+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~0BZ7KCL4NNEN.0 size=1048576 2023-12-03T11:48:40.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.560+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~0BZ7KCL4NNEN.1 size=248155 2023-12-03T11:48:40.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.562+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=0BZ7KCL4NNEN tag=ZX8DFJ3J296O type=write size=2345307 2023-12-03T11:48:40.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.563+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=J0K4W5DGJ26O type=del size=2279503 2023-12-03T11:48:40.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.564+0000 7f24574f5780 -1 < failed to complete operation key=SC8UOVLQQ0VD tag=S6SNG6MJGJS6 type=del size=4105109: (22) Invalid argument 2023-12-03T11:48:40.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.566+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.0 size=1048576 2023-12-03T11:48:40.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.567+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.1 size=1048576 2023-12-03T11:48:40.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.568+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.2 size=57735 2023-12-03T11:48:40.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.568+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=7IQFCBKMUV4J tag=FQD2ZF7HDT9A type=write size=3203463 2023-12-03T11:48:40.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.569+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=FHUN4105T37W type=del 2023-12-03T11:48:40.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.571+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=N02FHPB0O60Q type=del size=1174134 2023-12-03T11:48:40.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.572+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~2DMTWVFRFM2S.0 size=1048576 2023-12-03T11:48:40.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.573+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~2DMTWVFRFM2S.1 size=256764 2023-12-03T11:48:40.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.575+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=2DMTWVFRFM2S tag=OTX4Q481M0K7 type=write size=2353916 2023-12-03T11:48:40.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.576+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=S0H0AC28NLEA type=del size=606511 2023-12-03T11:48:40.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.577+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=OJI67QIO1F82 type=write size=819478 2023-12-03T11:48:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.578+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=PSZJ54EEFMXI type=del size=1741088 2023-12-03T11:48:40.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.579+0000 7f24574f5780 1 < canceled operation key=3EOEF9UMATIE tag=5GT32LHXKP10 type=del size=3623 2023-12-03T11:48:40.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.580+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=2RBIW5HDMALE type=del size=2166796 2023-12-03T11:48:40.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.582+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=OYWVN6FTA8O7 type=write size=690568 2023-12-03T11:48:40.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.583+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=RB42Y9T1XVQA type=del size=2648859 2023-12-03T11:48:40.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.584+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=FIE5TEC5RZHT type=del 2023-12-03T11:48:40.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.585+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=VE2LD9PCK2NH type=write size=552625 2023-12-03T11:48:40.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.585+0000 7f24574f5780 1 < canceled part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.0 size=1048576 2023-12-03T11:48:40.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.586+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.1 size=1048576 2023-12-03T11:48:40.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.587+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~GXG3MJRVTD67.2 size=765815 2023-12-03T11:48:40.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.589+0000 7f24574f5780 1 < canceled multipart upload key=1QR0TE9DXU5Z upload=GXG3MJRVTD67 tag=ZDP9MB33CN9H type=write size=3911543 2023-12-03T11:48:40.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.591+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~47SO3617WO9A.0 size=283146 2023-12-03T11:48:40.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.591+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=47SO3617WO9A tag=217TJQVCK8JV type=write size=1331722 2023-12-03T11:48:40.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.592+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~SK0M6P6FX5N8.0 size=1048576 2023-12-03T11:48:40.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.593+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~SK0M6P6FX5N8.1 size=736186 2023-12-03T11:48:40.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.595+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=SK0M6P6FX5N8 tag=COJM1RKRX8E0 type=write size=2833338 2023-12-03T11:48:40.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.598+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~VF364TH5XJX3.0 size=244438 2023-12-03T11:48:40.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.598+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=VF364TH5XJX3 tag=BSQH0MV5POCR type=write size=1293014 2023-12-03T11:48:40.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.599+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~0G5924C4DFNX.0 size=257895 2023-12-03T11:48:40.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.601+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=0G5924C4DFNX tag=P3ZOXPX3ORGE type=write size=1306471 2023-12-03T11:48:40.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.603+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~1LV4CFEH9E0E.0 size=1048576 2023-12-03T11:48:40.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.604+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~1LV4CFEH9E0E.1 size=259361 2023-12-03T11:48:40.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.604+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=1LV4CFEH9E0E tag=DHA7BO1AJIOR type=write size=2356513 2023-12-03T11:48:40.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.605+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=4YBCFHD4GEU4 type=del 2023-12-03T11:48:40.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.606+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=LVC6Z537IEGY type=write size=608867 2023-12-03T11:48:40.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.608+0000 7f24574f5780 1 < canceled operation key=3EOEF9UMATIE tag=MPT2LYPNG46S type=del size=4121615 2023-12-03T11:48:40.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.609+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=C0S2RAJKAIJK type=del size=1007305 2023-12-03T11:48:40.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.610+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=CR7XI6WIYZ6P type=del 2023-12-03T11:48:40.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.611+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=ASEHYVJ433UZ type=write size=476793 2023-12-03T11:48:40.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.613+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=745SW53HNQG0 type=del 2023-12-03T11:48:40.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.615+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.0 size=1048576 2023-12-03T11:48:40.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.616+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.1 size=1048576 2023-12-03T11:48:40.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.617+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.2 size=244439 2023-12-03T11:48:40.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.617+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=JQUI6T0IB11O tag=JIF8PNHHTAIW type=write size=3390167 2023-12-03T11:48:40.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.619+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=GH009D2WFCAC type=del 2023-12-03T11:48:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.622+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.0 size=1048576 2023-12-03T11:48:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.622+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.1 size=1048576 2023-12-03T11:48:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.623+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.2 size=128085 2023-12-03T11:48:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.623+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=SUU28M1B9GZ3 tag=0M57HP4V2SOT type=write size=3273813 2023-12-03T11:48:40.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.624+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=J0K4W5DGJ26O type=del 2023-12-03T11:48:40.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.626+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=76XQBPEB5BJX type=write size=658185 2023-12-03T11:48:40.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.627+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.0 size=1048576 2023-12-03T11:48:40.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.628+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.1 size=1048576 2023-12-03T11:48:40.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.629+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~7IQFCBKMUV4J.2 size=57735 2023-12-03T11:48:40.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.631+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=7IQFCBKMUV4J tag=FQD2ZF7HDT9A type=write size=3203463 2023-12-03T11:48:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.633+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=PWD2IVW5SR73 type=write size=556095 2023-12-03T11:48:40.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.634+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=N02FHPB0O60Q type=del 2023-12-03T11:48:40.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.635+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=WAQSLD3PMAR0 type=del size=153573 2023-12-03T11:48:40.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.637+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=S0H0AC28NLEA type=del size=606511 2023-12-03T11:48:40.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.639+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~782SI3JB2V3U.0 size=1048576 2023-12-03T11:48:40.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.640+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~782SI3JB2V3U.1 size=50021 2023-12-03T11:48:40.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.640+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=782SI3JB2V3U tag=9QFBZRK52ZUH type=write size=2147173 2023-12-03T11:48:40.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.643+0000 7f24574f5780 1 < suggested update operation key=PS1OYWI44L80 tag=PSZJ54EEFMXI type=del size=1741088 2023-12-03T11:48:40.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.644+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=5BD9FBJ1JRY6 type=del size=83537 2023-12-03T11:48:40.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.645+0000 7f24574f5780 1 < canceled operation key=9LKFY1U0JRBQ tag=2RBIW5HDMALE type=del size=2166796 2023-12-03T11:48:40.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.646+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=TJU2HLW5OT60 type=del size=2310388 2023-12-03T11:48:40.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.647+0000 7f24574f5780 -1 < failed to complete operation key=SUBILSL92D96 tag=RB42Y9T1XVQA type=del size=2648859: (22) Invalid argument 2023-12-03T11:48:40.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.648+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=IAP3VHVRN77C type=del size=3836049 2023-12-03T11:48:40.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.649+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=VE2LD9PCK2NH type=write size=552625 2023-12-03T11:48:40.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.651+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.0 size=1048576 2023-12-03T11:48:40.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.652+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.1 size=1048576 2023-12-03T11:48:40.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.653+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.2 size=76427 2023-12-03T11:48:40.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.653+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=4L25J8RP5FEF tag=875922DA8AIQ type=write size=3222155 2023-12-03T11:48:40.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.654+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~47SO3617WO9A.0 size=283146 2023-12-03T11:48:40.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.656+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=47SO3617WO9A tag=217TJQVCK8JV type=write size=1331722 2023-12-03T11:48:40.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.657+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=0GCL77P6QY80 type=del size=122689 2023-12-03T11:48:40.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.658+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~VF364TH5XJX3.0 size=244438 2023-12-03T11:48:40.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.660+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=VF364TH5XJX3 tag=BSQH0MV5POCR type=write size=1293014 2023-12-03T11:48:40.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.661+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=JTP4JU61B030 type=del size=795401 2023-12-03T11:48:40.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.662+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~1LV4CFEH9E0E.0 size=1048576 2023-12-03T11:48:40.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.663+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~1LV4CFEH9E0E.1 size=259361 2023-12-03T11:48:40.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.664+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=1LV4CFEH9E0E tag=DHA7BO1AJIOR type=write size=2356513 2023-12-03T11:48:40.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.667+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~ZH1U058D2784.0 size=151808 2023-12-03T11:48:40.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.667+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=ZH1U058D2784 tag=5VS6M8DSKLUL type=write size=1200384 2023-12-03T11:48:40.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.668+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=LVC6Z537IEGY type=write size=608867 2023-12-03T11:48:40.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.670+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.0 size=1048576 2023-12-03T11:48:40.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.671+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.1 size=1048576 2023-12-03T11:48:40.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.672+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.2 size=257935 2023-12-03T11:48:40.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.672+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=58LPKJ3Q4MXC tag=2170SWG7X35K type=write size=3403663 2023-12-03T11:48:40.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.674+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=C0S2RAJKAIJK type=del size=1007305 2023-12-03T11:48:40.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.675+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=I6AWXA51OMLR type=del size=1803002 2023-12-03T11:48:40.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.677+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=ASEHYVJ433UZ type=write size=476793 2023-12-03T11:48:40.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.678+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=P21UCS2KDEJ1 type=del size=1101082 2023-12-03T11:48:40.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.679+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.0 size=1048576 2023-12-03T11:48:40.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.679+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.1 size=1048576 2023-12-03T11:48:40.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.680+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~JQUI6T0IB11O.2 size=244439 2023-12-03T11:48:40.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.682+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=JQUI6T0IB11O tag=JIF8PNHHTAIW type=write size=3390167 2023-12-03T11:48:40.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.685+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~58GVSJF7GD68.0 size=430870 2023-12-03T11:48:40.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.685+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=58GVSJF7GD68 tag=0J19YBD3FP6T type=write size=1479446 2023-12-03T11:48:40.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.686+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.0 size=1048576 2023-12-03T11:48:40.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.687+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.1 size=1048576 2023-12-03T11:48:40.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.688+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~SUU28M1B9GZ3.2 size=128085 2023-12-03T11:48:40.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.690+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=SUU28M1B9GZ3 tag=0M57HP4V2SOT type=write size=3273813 2023-12-03T11:48:40.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.691+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=WTM8PPNH9JCT type=del size=2623853 2023-12-03T11:48:40.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.693+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=76XQBPEB5BJX type=write size=658185 2023-12-03T11:48:40.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.694+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=PU9D2EDXPM4C type=del size=1080826 2023-12-03T11:48:40.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.695+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=PWD2IVW5SR73 type=write size=556095 2023-12-03T11:48:40.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.697+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~QX5K808K0W9J.0 size=261675 2023-12-03T11:48:40.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.697+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=QX5K808K0W9J tag=CD6JMA1FR7K6 type=write size=1310251 2023-12-03T11:48:40.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.699+0000 7f24574f5780 1 < canceled operation key=MH1SRAATMNXB tag=WAQSLD3PMAR0 type=del size=153573 2023-12-03T11:48:40.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.701+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5Z6YSWTMCNJ3.0 size=1048576 2023-12-03T11:48:40.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.702+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5Z6YSWTMCNJ3.1 size=234575 2023-12-03T11:48:40.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.702+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=5Z6YSWTMCNJ3 tag=TG69G5D86G6F type=write size=2331727 2023-12-03T11:48:40.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.703+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~782SI3JB2V3U.0 size=1048576 2023-12-03T11:48:40.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.704+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~782SI3JB2V3U.1 size=50021 2023-12-03T11:48:40.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.706+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=782SI3JB2V3U tag=9QFBZRK52ZUH type=write size=2147173 2023-12-03T11:48:40.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.708+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~K96D96O0UGNW.0 size=541461 2023-12-03T11:48:40.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.708+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=K96D96O0UGNW tag=RLADIFNJMBV4 type=write size=1590037 2023-12-03T11:48:40.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.710+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=5BD9FBJ1JRY6 type=del 2023-12-03T11:48:40.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.711+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=3KP5SQK16BSP type=del size=944982 2023-12-03T11:48:40.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.714+0000 7f24574f5780 1 < suggested remove operation key=863EOZDYRWH3 tag=TJU2HLW5OT60 type=del 2023-12-03T11:48:40.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.715+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=8MMGAQAH2PLS type=del size=4016446 2023-12-03T11:48:40.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.716+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=IAP3VHVRN77C type=del 2023-12-03T11:48:40.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.717+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=AQFJ9MUG9SIY type=del size=4038562 2023-12-03T11:48:40.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.719+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.0 size=1048576 2023-12-03T11:48:40.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.720+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.1 size=1048576 2023-12-03T11:48:40.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.721+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~4L25J8RP5FEF.2 size=76427 2023-12-03T11:48:40.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.723+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=4L25J8RP5FEF tag=875922DA8AIQ type=write size=3222155 2023-12-03T11:48:40.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.725+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.0 size=1048576 2023-12-03T11:48:40.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.726+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.1 size=1048576 2023-12-03T11:48:40.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.727+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.2 size=991293 2023-12-03T11:48:40.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.727+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=JJW6CL7Q1SC9 tag=V11IGYBW8DDZ type=write size=4137021 2023-12-03T11:48:40.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.728+0000 7f24574f5780 1 < canceled operation key=BV8C0VSJ1GTW tag=0GCL77P6QY80 type=del size=122689 2023-12-03T11:48:40.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.730+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~XY6F3A1FC7XJ.0 size=1048576 2023-12-03T11:48:40.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.731+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~XY6F3A1FC7XJ.1 size=1029538 2023-12-03T11:48:40.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.731+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=XY6F3A1FC7XJ tag=Z8HSRVAK9P4C type=write size=3126690 2023-12-03T11:48:40.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.732+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=JTP4JU61B030 type=del 2023-12-03T11:48:40.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.733+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=3KFTCJ5YQ2WB type=del size=2400816 2023-12-03T11:48:40.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.735+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~ZH1U058D2784.0 size=151808 2023-12-03T11:48:40.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.736+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=ZH1U058D2784 tag=5VS6M8DSKLUL type=write size=1200384 2023-12-03T11:48:40.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.737+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=8WV4CXIVPFX0 type=write size=381619 2023-12-03T11:48:40.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.739+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.0 size=1048576 2023-12-03T11:48:40.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.740+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.1 size=1048576 2023-12-03T11:48:40.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.741+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~58LPKJ3Q4MXC.2 size=257935 2023-12-03T11:48:40.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.745+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=58LPKJ3Q4MXC tag=2170SWG7X35K type=write size=3403663 2023-12-03T11:48:40.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.747+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=YK1HIQJLDSZ8 type=del size=3710291 2023-12-03T11:48:40.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.748+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=I6AWXA51OMLR type=del 2023-12-03T11:48:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.750+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.0 size=1048576 2023-12-03T11:48:40.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.751+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.1 size=1048576 2023-12-03T11:48:40.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.751+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.2 size=659317 2023-12-03T11:48:40.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.751+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=CKZQJ0RXVENY tag=ZHZKSY0OMMUO type=write size=3805045 2023-12-03T11:48:40.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.753+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=P21UCS2KDEJ1 type=del 2023-12-03T11:48:40.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.755+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~0276ZHTJ1QVV.0 size=521134 2023-12-03T11:48:40.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.755+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=0276ZHTJ1QVV tag=2W7AAZTEDXU7 type=write size=1569710 2023-12-03T11:48:40.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.756+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~58GVSJF7GD68.0 size=430870 2023-12-03T11:48:40.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.758+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=58GVSJF7GD68 tag=0J19YBD3FP6T type=write size=1479446 2023-12-03T11:48:40.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.759+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=YCT4NTASVQB4 type=write size=678768 2023-12-03T11:48:40.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.760+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=WTM8PPNH9JCT type=del 2023-12-03T11:48:40.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.761+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=Y52CGXAVPL3P type=del size=1875686 2023-12-03T11:48:40.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.762+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=PU9D2EDXPM4C type=del 2023-12-03T11:48:40.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.763+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=A6JJ7H5KM2U9 type=del size=2091105 2023-12-03T11:48:40.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.764+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~QX5K808K0W9J.0 size=261675 2023-12-03T11:48:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.766+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=QX5K808K0W9J tag=CD6JMA1FR7K6 type=write size=1310251 2023-12-03T11:48:40.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.767+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=B7RDRITP9SH1 type=del size=228873 2023-12-03T11:48:40.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.769+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5Z6YSWTMCNJ3.0 size=1048576 2023-12-03T11:48:40.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.770+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5Z6YSWTMCNJ3.1 size=234575 2023-12-03T11:48:40.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.772+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=5Z6YSWTMCNJ3 tag=TG69G5D86G6F type=write size=2331727 2023-12-03T11:48:40.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.773+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=FNML1FQXEP98 type=del size=3228445 2023-12-03T11:48:40.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.774+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~K96D96O0UGNW.0 size=541461 2023-12-03T11:48:40.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.776+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=K96D96O0UGNW tag=RLADIFNJMBV4 type=write size=1590037 2023-12-03T11:48:40.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.778+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=1BBSERNJSOF9 type=del size=652950 2023-12-03T11:48:40.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.779+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=3KP5SQK16BSP type=del 2023-12-03T11:48:40.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.780+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=VHHYPYCR48CU type=write size=220395 2023-12-03T11:48:40.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.781+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=8MMGAQAH2PLS type=del 2023-12-03T11:48:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.782+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=MU1HQ0GZVQMS type=del size=2267817 2023-12-03T11:48:40.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.783+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=AQFJ9MUG9SIY type=del 2023-12-03T11:48:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.785+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~S1UQ73384W17.0 size=1048576 2023-12-03T11:48:40.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.787+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~S1UQ73384W17.1 size=447340 2023-12-03T11:48:40.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.787+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=S1UQ73384W17 tag=Y9R6FEPUQ9AS type=write size=2544492 2023-12-03T11:48:40.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.788+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.0 size=1048576 2023-12-03T11:48:40.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.789+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.1 size=1048576 2023-12-03T11:48:40.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.790+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~JJW6CL7Q1SC9.2 size=991293 2023-12-03T11:48:40.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.792+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=JJW6CL7Q1SC9 tag=V11IGYBW8DDZ type=write size=4137021 2023-12-03T11:48:40.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.794+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~13YR77SG3CTA.0 size=510551 2023-12-03T11:48:40.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.794+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=13YR77SG3CTA tag=287EA7F120X0 type=write size=1559127 2023-12-03T11:48:40.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.796+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~XY6F3A1FC7XJ.0 size=1048576 2023-12-03T11:48:40.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.797+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~XY6F3A1FC7XJ.1 size=1029538 2023-12-03T11:48:40.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.799+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=XY6F3A1FC7XJ tag=Z8HSRVAK9P4C type=write size=3126690 2023-12-03T11:48:40.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.801+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.0 size=1048576 2023-12-03T11:48:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.802+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.1 size=1048576 2023-12-03T11:48:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.802+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.2 size=168776 2023-12-03T11:48:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.802+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=IGX66KAQJ378 tag=3RM94PDORBB8 type=write size=3314504 2023-12-03T11:48:40.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.804+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=3KFTCJ5YQ2WB type=del 2023-12-03T11:48:40.806 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.805+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=MXVLYFPEEV5Z type=del size=1565481 2023-12-03T11:48:40.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.806+0000 7f24574f5780 1 < completed write operation key=PGTGVLK4DBN0 tag=8WV4CXIVPFX0 type=write size=381619 2023-12-03T11:48:40.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.807+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=4KS3VYMAGFO1 type=del size=3008825 2023-12-03T11:48:40.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.809+0000 7f24574f5780 1 < suggested update operation key=1QR0TE9DXU5Z tag=YK1HIQJLDSZ8 type=del size=3710291 2023-12-03T11:48:40.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.811+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.0 size=1048576 2023-12-03T11:48:40.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.812+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.1 size=1048576 2023-12-03T11:48:40.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.813+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.2 size=771255 2023-12-03T11:48:40.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.813+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=EQYGPP58P1P5 tag=1BR9XMIQR8XH type=write size=3916983 2023-12-03T11:48:40.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.813+0000 7f24574f5780 1 < canceled part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.0 size=1048576 2023-12-03T11:48:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.815+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.1 size=1048576 2023-12-03T11:48:40.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.815+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~CKZQJ0RXVENY.2 size=659317 2023-12-03T11:48:40.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.817+0000 7f24574f5780 1 < canceled multipart upload key=BV8C0VSJ1GTW upload=CKZQJ0RXVENY tag=ZHZKSY0OMMUO type=write size=3805045 2023-12-03T11:48:40.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.819+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=KG9ECP5ITSKO type=del size=1895685 2023-12-03T11:48:40.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.820+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~0276ZHTJ1QVV.0 size=521134 2023-12-03T11:48:40.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.822+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=0276ZHTJ1QVV tag=2W7AAZTEDXU7 type=write size=1569710 2023-12-03T11:48:40.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.823+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=HCKWW1PDOZ8L type=del size=1998819 2023-12-03T11:48:40.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.824+0000 7f24574f5780 1 < completed write operation key=6WY91DFIRUN8 tag=YCT4NTASVQB4 type=write size=678768 2023-12-03T11:48:40.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.825+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=UA1NT0UNWK4E type=del size=2357546 2023-12-03T11:48:40.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.826+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=Y52CGXAVPL3P type=del 2023-12-03T11:48:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.828+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~SH3YJW85X5AM.0 size=753497 2023-12-03T11:48:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.828+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=SH3YJW85X5AM tag=BKG8YKEJ6QAH type=write size=1802073 2023-12-03T11:48:40.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.830+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=A6JJ7H5KM2U9 type=del 2023-12-03T11:48:40.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.832+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.0 size=1048576 2023-12-03T11:48:40.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.833+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.1 size=1048576 2023-12-03T11:48:40.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.834+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.2 size=290621 2023-12-03T11:48:40.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.834+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=HLR4XWU10CZS tag=HOB0WDWHJ9TL type=write size=3436349 2023-12-03T11:48:40.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.835+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=B7RDRITP9SH1 type=del 2023-12-03T11:48:40.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.836+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=UHT14JSZK7PN type=del size=3047397 2023-12-03T11:48:40.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.837+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=FNML1FQXEP98 type=del 2023-12-03T11:48:40.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.838+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=U375TZIJ36MP type=write size=143469 2023-12-03T11:48:40.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.840+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=1BBSERNJSOF9 type=del 2023-12-03T11:48:40.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.842+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.0 size=1048576 2023-12-03T11:48:40.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.843+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.1 size=1048576 2023-12-03T11:48:40.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.844+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.2 size=673579 2023-12-03T11:48:40.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.844+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=FXSZ3WDRMDOX tag=FO83B9XJKUQ9 type=write size=3819307 2023-12-03T11:48:40.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.845+0000 7f24574f5780 1 < completed write operation key=BV8C0VSJ1GTW tag=VHHYPYCR48CU type=write size=220395 2023-12-03T11:48:40.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.847+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.0 size=1048576 2023-12-03T11:48:40.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.848+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.1 size=1048576 2023-12-03T11:48:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.849+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.2 size=177103 2023-12-03T11:48:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.849+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=HSVOXJRXJ4Q4 tag=6J40ZZEO6S2K type=write size=3322831 2023-12-03T11:48:40.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.851+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=MU1HQ0GZVQMS type=del 2023-12-03T11:48:40.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.853+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~B3T2JA6RQF90.0 size=790511 2023-12-03T11:48:40.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.853+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=B3T2JA6RQF90 tag=GUG4CSH7V0J1 type=write size=1839087 2023-12-03T11:48:40.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.853+0000 7f24574f5780 1 < canceled part key=_multipart_XRHVYVADRWET.2~S1UQ73384W17.0 size=1048576 2023-12-03T11:48:40.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.854+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~S1UQ73384W17.1 size=447340 2023-12-03T11:48:40.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.856+0000 7f24574f5780 1 < canceled multipart upload key=XRHVYVADRWET upload=S1UQ73384W17 tag=Y9R6FEPUQ9AS type=write size=2544492 2023-12-03T11:48:40.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.858+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.0 size=1048576 2023-12-03T11:48:40.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.859+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.1 size=1048576 2023-12-03T11:48:40.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.860+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.2 size=142607 2023-12-03T11:48:40.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.860+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=XQN2RPXD574E tag=7XCEIZB2ASBS type=write size=3288335 2023-12-03T11:48:40.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.861+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~13YR77SG3CTA.0 size=510551 2023-12-03T11:48:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.863+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=13YR77SG3CTA tag=287EA7F120X0 type=write size=1559127 2023-12-03T11:48:40.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.865+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~Z2NEXURLLQ00.0 size=1048576 2023-12-03T11:48:40.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.866+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~Z2NEXURLLQ00.1 size=822966 2023-12-03T11:48:40.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.866+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=Z2NEXURLLQ00 tag=A5ILFFY5AGA0 type=write size=2920118 2023-12-03T11:48:40.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.867+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.0 size=1048576 2023-12-03T11:48:40.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.868+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.1 size=1048576 2023-12-03T11:48:40.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.869+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~IGX66KAQJ378.2 size=168776 2023-12-03T11:48:40.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.871+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=IGX66KAQJ378 tag=3RM94PDORBB8 type=write size=3314504 2023-12-03T11:48:40.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.872+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=SZQXKG0AJEKR type=write size=990542 2023-12-03T11:48:40.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.873+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=MXVLYFPEEV5Z type=del 2023-12-03T11:48:40.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.874+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=ISOTCY671BVN type=del size=1303158 2023-12-03T11:48:40.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.875+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=4KS3VYMAGFO1 type=del 2023-12-03T11:48:40.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.876+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=17BS1OGIERWT type=del size=2343237 2023-12-03T11:48:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.877+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.0 size=1048576 2023-12-03T11:48:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.878+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.1 size=1048576 2023-12-03T11:48:40.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.879+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~EQYGPP58P1P5.2 size=771255 2023-12-03T11:48:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.881+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=EQYGPP58P1P5 tag=1BR9XMIQR8XH type=write size=3916983 2023-12-03T11:48:40.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.883+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.0 size=1048576 2023-12-03T11:48:40.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.884+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.1 size=1048576 2023-12-03T11:48:40.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.885+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.2 size=257855 2023-12-03T11:48:40.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.885+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=2335OWS7AK87 tag=YBOGEE5Z77ZW type=write size=3403583 2023-12-03T11:48:40.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.886+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=KG9ECP5ITSKO type=del 2023-12-03T11:48:40.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.888+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~TR9TEBLQAGYC.0 size=844341 2023-12-03T11:48:40.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.888+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=TR9TEBLQAGYC tag=XGV9C2VDWFXV type=write size=1892917 2023-12-03T11:48:40.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.890+0000 7f24574f5780 1 < suggested remove operation key=MY3K9M4QRS3L tag=HCKWW1PDOZ8L type=del 2023-12-03T11:48:40.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.892+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~SLM8IBBV5U7Q.0 size=1048576 2023-12-03T11:48:40.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.893+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~SLM8IBBV5U7Q.1 size=139783 2023-12-03T11:48:40.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.893+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=SLM8IBBV5U7Q tag=MJJSW7J2L1VC type=write size=2236935 2023-12-03T11:48:40.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.895+0000 7f24574f5780 1 < suggested update operation key=XZFHFS76WNR4 tag=UA1NT0UNWK4E type=del size=2357546 2023-12-03T11:48:40.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.897+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=VD5I6LN8MMLP type=del size=1142569 2023-12-03T11:48:40.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.898+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~SH3YJW85X5AM.0 size=753497 2023-12-03T11:48:40.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.900+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=SH3YJW85X5AM tag=BKG8YKEJ6QAH type=write size=1802073 2023-12-03T11:48:40.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.902+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.0 size=1048576 2023-12-03T11:48:40.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.903+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.1 size=1048576 2023-12-03T11:48:40.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.904+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.2 size=893283 2023-12-03T11:48:40.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.904+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=K09DXMHZIG8S tag=5AEM8ZFVKG0V type=write size=4039011 2023-12-03T11:48:40.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.905+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.0 size=1048576 2023-12-03T11:48:40.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.906+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.1 size=1048576 2023-12-03T11:48:40.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.907+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~HLR4XWU10CZS.2 size=290621 2023-12-03T11:48:40.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.909+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=HLR4XWU10CZS tag=HOB0WDWHJ9TL type=write size=3436349 2023-12-03T11:48:40.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.911+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=77NA3MNEJZHN type=del size=3287753 2023-12-03T11:48:40.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.912+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=UHT14JSZK7PN type=del 2023-12-03T11:48:40.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.913+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=8MXFHTFH698S type=del size=1718500 2023-12-03T11:48:40.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.914+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=U375TZIJ36MP type=write size=143469 2023-12-03T11:48:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.916+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.0 size=1048576 2023-12-03T11:48:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.917+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.1 size=1048576 2023-12-03T11:48:40.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.918+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.2 size=70144 2023-12-03T11:48:40.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.918+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=RZIS6EQ1G1LV tag=EKYXS3OD25GY type=write size=3215872 2023-12-03T11:48:40.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.919+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.0 size=1048576 2023-12-03T11:48:40.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.921+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.1 size=1048576 2023-12-03T11:48:40.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.921+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~FXSZ3WDRMDOX.2 size=673579 2023-12-03T11:48:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.923+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=FXSZ3WDRMDOX tag=FO83B9XJKUQ9 type=write size=3819307 2023-12-03T11:48:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.924+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=SDGUWF6Y1C1I type=del size=69123 2023-12-03T11:48:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.925+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.0 size=1048576 2023-12-03T11:48:40.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.926+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.1 size=1048576 2023-12-03T11:48:40.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.927+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~HSVOXJRXJ4Q4.2 size=177103 2023-12-03T11:48:40.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.929+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=HSVOXJRXJ4Q4 tag=6J40ZZEO6S2K type=write size=3322831 2023-12-03T11:48:40.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.930+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=3JBIQ54M4L5R type=del size=505806 2023-12-03T11:48:40.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.931+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~B3T2JA6RQF90.0 size=790511 2023-12-03T11:48:40.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.933+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=B3T2JA6RQF90 tag=GUG4CSH7V0J1 type=write size=1839087 2023-12-03T11:48:40.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.935+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=84TVUSJP1LKU type=del size=313899 2023-12-03T11:48:40.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.936+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.0 size=1048576 2023-12-03T11:48:40.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.937+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.1 size=1048576 2023-12-03T11:48:40.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.938+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~XQN2RPXD574E.2 size=142607 2023-12-03T11:48:40.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.941+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=XQN2RPXD574E tag=7XCEIZB2ASBS type=write size=3288335 2023-12-03T11:48:40.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.943+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=Y83MT2ITQ178 type=del size=4189263 2023-12-03T11:48:40.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.943+0000 7f24574f5780 1 < canceled part key=_multipart_BV8C0VSJ1GTW.2~Z2NEXURLLQ00.0 size=1048576 2023-12-03T11:48:40.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.944+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~Z2NEXURLLQ00.1 size=822966 2023-12-03T11:48:40.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.946+0000 7f24574f5780 1 < canceled multipart upload key=BV8C0VSJ1GTW upload=Z2NEXURLLQ00 tag=A5ILFFY5AGA0 type=write size=2920118 2023-12-03T11:48:40.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.947+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=PPH8Z8BSPNV6 type=del size=63062 2023-12-03T11:48:40.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.950+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=SZQXKG0AJEKR type=write size=990542 2023-12-03T11:48:40.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.952+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~3V9GQONRY0XR.0 size=1048576 2023-12-03T11:48:40.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.953+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~3V9GQONRY0XR.1 size=408949 2023-12-03T11:48:40.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.953+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=3V9GQONRY0XR tag=1W6YU7YOLNEI type=write size=2506101 2023-12-03T11:48:40.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.955+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=ISOTCY671BVN type=del 2023-12-03T11:48:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.957+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.0 size=1048576 2023-12-03T11:48:40.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.958+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.1 size=1048576 2023-12-03T11:48:40.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.960+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.2 size=1034992 2023-12-03T11:48:40.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.960+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=KGRIOCFNQ6B3 tag=DF2BDQ7LTQB9 type=write size=4180720 2023-12-03T11:48:40.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.961+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=17BS1OGIERWT type=del 2023-12-03T11:48:40.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.963+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.0 size=1048576 2023-12-03T11:48:40.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.964+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.1 size=1048576 2023-12-03T11:48:40.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.965+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.2 size=1009624 2023-12-03T11:48:40.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.965+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=3VQ6S3Y6XPA2 tag=76EHUQK3M6IW type=write size=4155352 2023-12-03T11:48:40.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.966+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.0 size=1048576 2023-12-03T11:48:40.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.967+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.1 size=1048576 2023-12-03T11:48:40.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.968+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~2335OWS7AK87.2 size=257855 2023-12-03T11:48:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.970+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=2335OWS7AK87 tag=YBOGEE5Z77ZW type=write size=3403583 2023-12-03T11:48:40.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.972+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~HQU9DNNO7LYP.0 size=50878 2023-12-03T11:48:40.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.972+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=HQU9DNNO7LYP tag=VW8P3B79L06P type=write size=1099454 2023-12-03T11:48:40.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.972+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~TR9TEBLQAGYC.0 size=844341 2023-12-03T11:48:40.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.974+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=TR9TEBLQAGYC tag=XGV9C2VDWFXV type=write size=1892917 2023-12-03T11:48:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.975+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=BV7J4611BJVF type=del size=852006 2023-12-03T11:48:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.976+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~SLM8IBBV5U7Q.0 size=1048576 2023-12-03T11:48:40.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.977+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~SLM8IBBV5U7Q.1 size=139783 2023-12-03T11:48:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.979+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=SLM8IBBV5U7Q tag=MJJSW7J2L1VC type=write size=2236935 2023-12-03T11:48:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.980+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=YRBSWJ6KQ7E1 type=del size=1878794 2023-12-03T11:48:40.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.981+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=VD5I6LN8MMLP type=del 2023-12-03T11:48:40.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.982+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=EHKPJXXKJV40 type=del size=845818 2023-12-03T11:48:40.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.983+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.0 size=1048576 2023-12-03T11:48:40.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.984+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.1 size=1048576 2023-12-03T11:48:40.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.985+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~K09DXMHZIG8S.2 size=893283 2023-12-03T11:48:40.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.986+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=K09DXMHZIG8S tag=5AEM8ZFVKG0V type=write size=4039011 2023-12-03T11:48:40.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.987+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=3CCGQJGCBLZY type=write size=445354 2023-12-03T11:48:40.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.989+0000 7f24574f5780 1 < canceled operation key=1QR0TE9DXU5Z tag=77NA3MNEJZHN type=del size=3287753 2023-12-03T11:48:40.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.990+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=GRM3HIZBFQ76 type=del size=1324765 2023-12-03T11:48:40.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.991+0000 7f24574f5780 1 < canceled operation key=WZMWCSAZX8RB tag=8MXFHTFH698S type=del size=1718500 2023-12-03T11:48:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.992+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=DJYLXY61YD2X type=write size=544079 2023-12-03T11:48:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.993+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.0 size=1048576 2023-12-03T11:48:40.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.994+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.1 size=1048576 2023-12-03T11:48:40.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.995+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~RZIS6EQ1G1LV.2 size=70144 2023-12-03T11:48:40.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.997+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=RZIS6EQ1G1LV tag=EKYXS3OD25GY type=write size=3215872 2023-12-03T11:48:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:40.999+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.0 size=1048576 2023-12-03T11:48:41.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.000+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.1 size=1048576 2023-12-03T11:48:41.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.001+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.2 size=270860 2023-12-03T11:48:41.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.001+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=9IN30G20JJI6 tag=EO1NMCGMZ58J type=write size=3416588 2023-12-03T11:48:41.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.002+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=SDGUWF6Y1C1I type=del 2023-12-03T11:48:41.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.004+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.0 size=1048576 2023-12-03T11:48:41.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.005+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.1 size=1048576 2023-12-03T11:48:41.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.006+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.2 size=177955 2023-12-03T11:48:41.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.006+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=T2NIVJUE5E2H tag=H5SA7868DVSP type=write size=3323683 2023-12-03T11:48:41.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.007+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=3JBIQ54M4L5R type=del 2023-12-03T11:48:41.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.008+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=P9TO1F3J8A01 type=del size=1474996 2023-12-03T11:48:41.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.009+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=84TVUSJP1LKU type=del 2023-12-03T11:48:41.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.010+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=Y9PVGX6MD827 type=del size=1163365 2023-12-03T11:48:41.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.012+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=Y83MT2ITQ178 type=del 2023-12-03T11:48:41.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.014+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~YG90BTI9HUFH.0 size=1048576 2023-12-03T11:48:41.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.015+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~YG90BTI9HUFH.1 size=443636 2023-12-03T11:48:41.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.015+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=YG90BTI9HUFH tag=BDJ30L8Y5EUU type=write size=2540788 2023-12-03T11:48:41.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.016+0000 7f24574f5780 -1 < failed to complete operation key=S73M6UFMX5NH tag=PPH8Z8BSPNV6 type=del size=63062: (22) Invalid argument 2023-12-03T11:48:41.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.018+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~X3REF9RCXMIS.0 size=1048576 2023-12-03T11:48:41.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.019+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~X3REF9RCXMIS.1 size=415515 2023-12-03T11:48:41.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.019+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=X3REF9RCXMIS tag=VNRGBFMSS83V type=write size=2512667 2023-12-03T11:48:41.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.020+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~3V9GQONRY0XR.0 size=1048576 2023-12-03T11:48:41.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.021+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~3V9GQONRY0XR.1 size=408949 2023-12-03T11:48:41.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.023+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=3V9GQONRY0XR tag=1W6YU7YOLNEI type=write size=2506101 2023-12-03T11:48:41.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.025+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~OFXYQZMNHEIG.0 size=824863 2023-12-03T11:48:41.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.025+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=OFXYQZMNHEIG tag=BOPB8KDMMW9G type=write size=1873439 2023-12-03T11:48:41.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.026+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.0 size=1048576 2023-12-03T11:48:41.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.027+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.1 size=1048576 2023-12-03T11:48:41.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.028+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~KGRIOCFNQ6B3.2 size=1034992 2023-12-03T11:48:41.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.030+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=KGRIOCFNQ6B3 tag=DF2BDQ7LTQB9 type=write size=4180720 2023-12-03T11:48:41.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.032+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=73YN3XC6VTDP type=del size=4008913 2023-12-03T11:48:41.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.033+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.0 size=1048576 2023-12-03T11:48:41.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.034+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.1 size=1048576 2023-12-03T11:48:41.036 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.035+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~3VQ6S3Y6XPA2.2 size=1009624 2023-12-03T11:48:41.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.038+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=3VQ6S3Y6XPA2 tag=76EHUQK3M6IW type=write size=4155352 2023-12-03T11:48:41.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.041+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=J367X8PPIWZV type=write size=588747 2023-12-03T11:48:41.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.042+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~HQU9DNNO7LYP.0 size=50878 2023-12-03T11:48:41.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.044+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=HQU9DNNO7LYP tag=VW8P3B79L06P type=write size=1099454 2023-12-03T11:48:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.045+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=P306DZ3U554F type=del size=2744599 2023-12-03T11:48:41.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.046+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=BV7J4611BJVF type=del 2023-12-03T11:48:41.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.048+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=65VU8C1PRMD1 type=del size=2506442 2023-12-03T11:48:41.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.049+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=YRBSWJ6KQ7E1 type=del 2023-12-03T11:48:41.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.050+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=ZB15YF21NYOP type=del size=462974 2023-12-03T11:48:41.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.051+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=EHKPJXXKJV40 type=del 2023-12-03T11:48:41.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.052+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=VLH26YS828EZ type=del size=658786 2023-12-03T11:48:41.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.053+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=3CCGQJGCBLZY type=write size=445354 2023-12-03T11:48:41.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.055+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~RD7FRLXMZGIL.0 size=541769 2023-12-03T11:48:41.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.055+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=RD7FRLXMZGIL tag=XPSBSJOT6I67 type=write size=1590345 2023-12-03T11:48:41.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.059+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=GRM3HIZBFQ76 type=del 2023-12-03T11:48:41.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.061+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.0 size=1048576 2023-12-03T11:48:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.062+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.1 size=1048576 2023-12-03T11:48:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.063+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.2 size=886619 2023-12-03T11:48:41.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.063+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=BQIFJJX5PVWE tag=1G5QDCPOMVBD type=write size=4032347 2023-12-03T11:48:41.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.065+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=DJYLXY61YD2X type=write size=544079 2023-12-03T11:48:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.066+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=RUWBGMCWH2PC type=del size=1258546 2023-12-03T11:48:41.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.067+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.0 size=1048576 2023-12-03T11:48:41.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.068+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.1 size=1048576 2023-12-03T11:48:41.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.070+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~9IN30G20JJI6.2 size=270860 2023-12-03T11:48:41.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.072+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=9IN30G20JJI6 tag=EO1NMCGMZ58J type=write size=3416588 2023-12-03T11:48:41.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.074+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~S1ZSSZMWSC40.0 size=1048576 2023-12-03T11:48:41.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.075+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~S1ZSSZMWSC40.1 size=305255 2023-12-03T11:48:41.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.075+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=S1ZSSZMWSC40 tag=2K282WH2TR56 type=write size=2402407 2023-12-03T11:48:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.076+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.0 size=1048576 2023-12-03T11:48:41.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.077+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.1 size=1048576 2023-12-03T11:48:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.078+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~T2NIVJUE5E2H.2 size=177955 2023-12-03T11:48:41.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.080+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=T2NIVJUE5E2H tag=H5SA7868DVSP type=write size=3323683 2023-12-03T11:48:41.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.082+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~94JO5P94WIEL.0 size=90538 2023-12-03T11:48:41.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.082+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=94JO5P94WIEL tag=VFMSKUWCLG9F type=write size=1139114 2023-12-03T11:48:41.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.083+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=P9TO1F3J8A01 type=del 2023-12-03T11:48:41.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.085+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.0 size=1048576 2023-12-03T11:48:41.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.086+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.1 size=1048576 2023-12-03T11:48:41.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.087+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.2 size=246448 2023-12-03T11:48:41.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.087+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=Y68RCVWQ5AQF tag=MIIHHRR3O60I type=write size=3392176 2023-12-03T11:48:41.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.090+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=Y9PVGX6MD827 type=del 2023-12-03T11:48:41.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.091+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=O4FWISRRIUMB type=del size=1762312 2023-12-03T11:48:41.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.092+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~YG90BTI9HUFH.0 size=1048576 2023-12-03T11:48:41.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.092+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~YG90BTI9HUFH.1 size=443636 2023-12-03T11:48:41.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.094+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=YG90BTI9HUFH tag=BDJ30L8Y5EUU type=write size=2540788 2023-12-03T11:48:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.095+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=4E2CXT29LMUN type=del size=826392 2023-12-03T11:48:41.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.096+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~X3REF9RCXMIS.0 size=1048576 2023-12-03T11:48:41.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.096+0000 7f24574f5780 1 < canceled part key=_multipart_1QR0TE9DXU5Z.2~X3REF9RCXMIS.1 size=415515 2023-12-03T11:48:41.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.098+0000 7f24574f5780 1 < canceled multipart upload key=1QR0TE9DXU5Z upload=X3REF9RCXMIS tag=VNRGBFMSS83V type=write size=2512667 2023-12-03T11:48:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.100+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~RMFOXIMOVA82.0 size=252282 2023-12-03T11:48:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.100+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=RMFOXIMOVA82 tag=YZJ5IZIG1U5E type=write size=1300858 2023-12-03T11:48:41.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.101+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~OFXYQZMNHEIG.0 size=824863 2023-12-03T11:48:41.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.103+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=OFXYQZMNHEIG tag=BOPB8KDMMW9G type=write size=1873439 2023-12-03T11:48:41.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.104+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=UTKNUXG29GFZ type=del size=4156713 2023-12-03T11:48:41.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.105+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=73YN3XC6VTDP type=del 2023-12-03T11:48:41.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.106+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=YF1OFKLBRIBE type=del size=2235822 2023-12-03T11:48:41.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.107+0000 7f24574f5780 1 < completed write operation key=1QR0TE9DXU5Z tag=J367X8PPIWZV type=write size=588747 2023-12-03T11:48:41.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.108+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=RCHSRDVRMD41 type=write size=328814 2023-12-03T11:48:41.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.110+0000 7f24574f5780 1 < canceled operation key=3EOEF9UMATIE tag=P306DZ3U554F type=del size=2744599 2023-12-03T11:48:41.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.112+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~6IPFLSTMA9X6.0 size=1048576 2023-12-03T11:48:41.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.113+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~6IPFLSTMA9X6.1 size=328377 2023-12-03T11:48:41.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.113+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=6IPFLSTMA9X6 tag=VE723BHBVICO type=write size=2425529 2023-12-03T11:48:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.114+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=65VU8C1PRMD1 type=del 2023-12-03T11:48:41.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.115+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=0NLVYUAS1RNG type=del size=1875443 2023-12-03T11:48:41.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.116+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=ZB15YF21NYOP type=del 2023-12-03T11:48:41.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.117+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=IYPG3YJGFHDG type=del size=64205 2023-12-03T11:48:41.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.118+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=VLH26YS828EZ type=del 2023-12-03T11:48:41.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.119+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=VP3S8XV67USN type=del size=1534265 2023-12-03T11:48:41.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.121+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~RD7FRLXMZGIL.0 size=541769 2023-12-03T11:48:41.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.123+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=RD7FRLXMZGIL tag=XPSBSJOT6I67 type=write size=1590345 2023-12-03T11:48:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.125+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.0 size=1048576 2023-12-03T11:48:41.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.126+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.1 size=1048576 2023-12-03T11:48:41.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.127+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.2 size=180609 2023-12-03T11:48:41.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.127+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=BW8QUVV4QFZ2 tag=W40H2F52IYFI type=write size=3326337 2023-12-03T11:48:41.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.129+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.0 size=1048576 2023-12-03T11:48:41.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.130+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.1 size=1048576 2023-12-03T11:48:41.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.131+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~BQIFJJX5PVWE.2 size=886619 2023-12-03T11:48:41.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.133+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=BQIFJJX5PVWE tag=1G5QDCPOMVBD type=write size=4032347 2023-12-03T11:48:41.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.134+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=FUBC2ZI8RKFW type=del size=3912232 2023-12-03T11:48:41.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.135+0000 7f24574f5780 1 < canceled operation key=HM9JHOV8W24F tag=RUWBGMCWH2PC type=del size=1258546 2023-12-03T11:48:41.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.137+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=NGDKISGPNNWK type=del size=1022480 2023-12-03T11:48:41.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.138+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~S1ZSSZMWSC40.0 size=1048576 2023-12-03T11:48:41.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.139+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~S1ZSSZMWSC40.1 size=305255 2023-12-03T11:48:41.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.141+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=S1ZSSZMWSC40 tag=2K282WH2TR56 type=write size=2402407 2023-12-03T11:48:41.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.142+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=52Q6RXM9Z1NC type=del size=3043548 2023-12-03T11:48:41.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.143+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~94JO5P94WIEL.0 size=90538 2023-12-03T11:48:41.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.145+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=94JO5P94WIEL tag=VFMSKUWCLG9F type=write size=1139114 2023-12-03T11:48:41.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.146+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=PDTEDIQOFITH type=del size=2775701 2023-12-03T11:48:41.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.147+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.0 size=1048576 2023-12-03T11:48:41.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.148+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.1 size=1048576 2023-12-03T11:48:41.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.149+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~Y68RCVWQ5AQF.2 size=246448 2023-12-03T11:48:41.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.151+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=Y68RCVWQ5AQF tag=MIIHHRR3O60I type=write size=3392176 2023-12-03T11:48:41.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.153+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~ZFKA3U5F3HNZ.0 size=620607 2023-12-03T11:48:41.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.153+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=ZFKA3U5F3HNZ tag=UCG9NQDELD7Y type=write size=1669183 2023-12-03T11:48:41.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.154+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=O4FWISRRIUMB type=del 2023-12-03T11:48:41.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.156+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~LO88UNMG0AFC.0 size=508272 2023-12-03T11:48:41.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.156+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=LO88UNMG0AFC tag=6XQ31GW2KPPF type=write size=1556848 2023-12-03T11:48:41.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.158+0000 7f24574f5780 1 < suggested remove operation key=SC8UOVLQQ0VD tag=4E2CXT29LMUN type=del 2023-12-03T11:48:41.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.160+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=ZFRTMFTS1IYK type=del size=116801 2023-12-03T11:48:41.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.161+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~RMFOXIMOVA82.0 size=252282 2023-12-03T11:48:41.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.163+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=RMFOXIMOVA82 tag=YZJ5IZIG1U5E type=write size=1300858 2023-12-03T11:48:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.165+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~BC4E0H1P4JHQ.0 size=71146 2023-12-03T11:48:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.165+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=BC4E0H1P4JHQ tag=G63TJAJQ2K5A type=write size=1119722 2023-12-03T11:48:41.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.166+0000 7f24574f5780 1 < canceled operation key=SUBILSL92D96 tag=UTKNUXG29GFZ type=del size=4156713 2023-12-03T11:48:41.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.167+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=27CHBKC0MS9M type=write size=136864 2023-12-03T11:48:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.168+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=YF1OFKLBRIBE type=del 2023-12-03T11:48:41.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.170+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~JDB2AF772GYY.0 size=1048576 2023-12-03T11:48:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.171+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~JDB2AF772GYY.1 size=807758 2023-12-03T11:48:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.171+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=JDB2AF772GYY tag=MJKM0VFIILB2 type=write size=2904910 2023-12-03T11:48:41.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.172+0000 7f24574f5780 -1 < failed to complete operation key=SC8UOVLQQ0VD tag=RCHSRDVRMD41 type=write size=328814: (22) Invalid argument 2023-12-03T11:48:41.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.173+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=KH6K293QRLAV type=del size=3286342 2023-12-03T11:48:41.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.175+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~6IPFLSTMA9X6.0 size=1048576 2023-12-03T11:48:41.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.176+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~6IPFLSTMA9X6.1 size=328377 2023-12-03T11:48:41.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.179+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=6IPFLSTMA9X6 tag=VE723BHBVICO type=write size=2425529 2023-12-03T11:48:41.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.181+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=WOXB7CJLBMRF type=del size=2759286 2023-12-03T11:48:41.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.183+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=0NLVYUAS1RNG type=del size=1875443 2023-12-03T11:48:41.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.186+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~ZBLB0QVIAOXA.0 size=1048576 2023-12-03T11:48:41.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.186+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~ZBLB0QVIAOXA.1 size=219780 2023-12-03T11:48:41.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.187+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=ZBLB0QVIAOXA tag=DLVHFZ289UVV type=write size=2316932 2023-12-03T11:48:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.188+0000 7f24574f5780 1 < canceled operation key=BV8C0VSJ1GTW tag=IYPG3YJGFHDG type=del size=64205 2023-12-03T11:48:41.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.189+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=2S1M54GII6C9 type=del size=4192846 2023-12-03T11:48:41.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.190+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=VP3S8XV67USN type=del 2023-12-03T11:48:41.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.192+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.0 size=1048576 2023-12-03T11:48:41.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.193+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.1 size=1048576 2023-12-03T11:48:41.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.194+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.2 size=974493 2023-12-03T11:48:41.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.194+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=3O32ST39SO0H tag=C2R7LXC56TKU type=write size=4120221 2023-12-03T11:48:41.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.195+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.0 size=1048576 2023-12-03T11:48:41.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.196+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.1 size=1048576 2023-12-03T11:48:41.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.197+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~BW8QUVV4QFZ2.2 size=180609 2023-12-03T11:48:41.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.199+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=BW8QUVV4QFZ2 tag=W40H2F52IYFI type=write size=3326337 2023-12-03T11:48:41.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.200+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=I0XTUSBRGHKU type=write size=154478 2023-12-03T11:48:41.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.201+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=FUBC2ZI8RKFW type=del 2023-12-03T11:48:41.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.202+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=K5JP3IV6RHJD type=write size=412264 2023-12-03T11:48:41.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.204+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=NGDKISGPNNWK type=del 2023-12-03T11:48:41.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.205+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=PQAPT47Y7A5V type=write size=612044 2023-12-03T11:48:41.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.206+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=52Q6RXM9Z1NC type=del 2023-12-03T11:48:41.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.207+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=5JLDTF1BKJ89 type=del size=2369952 2023-12-03T11:48:41.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.208+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=PDTEDIQOFITH type=del size=2775701 2023-12-03T11:48:41.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.209+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=VXXMQFIHZ1MY type=del size=4142652 2023-12-03T11:48:41.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.211+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~ZFKA3U5F3HNZ.0 size=620607 2023-12-03T11:48:41.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.212+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=ZFKA3U5F3HNZ tag=UCG9NQDELD7Y type=write size=1669183 2023-12-03T11:48:41.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.215+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~2SAT23YY4V15.0 size=829087 2023-12-03T11:48:41.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.215+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=2SAT23YY4V15 tag=U9VPQ3IO5JYO type=write size=1877663 2023-12-03T11:48:41.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.216+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~LO88UNMG0AFC.0 size=508272 2023-12-03T11:48:41.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.218+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=LO88UNMG0AFC tag=6XQ31GW2KPPF type=write size=1556848 2023-12-03T11:48:41.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.219+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=X1U93LLXQKWG type=del size=2571178 2023-12-03T11:48:41.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.221+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=ZFRTMFTS1IYK type=del 2023-12-03T11:48:41.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.222+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=3GI3X3N9WRLW type=del size=2885508 2023-12-03T11:48:41.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.223+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~BC4E0H1P4JHQ.0 size=71146 2023-12-03T11:48:41.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.225+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=BC4E0H1P4JHQ tag=G63TJAJQ2K5A type=write size=1119722 2023-12-03T11:48:41.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.227+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=W1OMOM8UP1OP type=del size=3777413 2023-12-03T11:48:41.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.228+0000 7f24574f5780 1 < completed write operation key=3EOEF9UMATIE tag=27CHBKC0MS9M type=write size=136864 2023-12-03T11:48:41.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.229+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=OTGGDCFZQNM9 type=del size=3916152 2023-12-03T11:48:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.230+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~JDB2AF772GYY.0 size=1048576 2023-12-03T11:48:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.230+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~JDB2AF772GYY.1 size=807758 2023-12-03T11:48:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.233+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=JDB2AF772GYY tag=MJKM0VFIILB2 type=write size=2904910 2023-12-03T11:48:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.234+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=4CMSLRPIJGMT type=del size=3519552 2023-12-03T11:48:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.235+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=KH6K293QRLAV type=del 2023-12-03T11:48:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.236+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=MQ9Z08223CZZ type=del size=1382769 2023-12-03T11:48:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.238+0000 7f24574f5780 1 < canceled operation key=QVLPCWC4T8N8 tag=WOXB7CJLBMRF type=del size=2759286 2023-12-03T11:48:41.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.240+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.0 size=1048576 2023-12-03T11:48:41.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.241+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.1 size=1048576 2023-12-03T11:48:41.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.242+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.2 size=1039669 2023-12-03T11:48:41.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.242+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=L6FOM1XNZ0EA tag=8XXS1XXTQILD type=write size=4185397 2023-12-03T11:48:41.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.243+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~ZBLB0QVIAOXA.0 size=1048576 2023-12-03T11:48:41.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.243+0000 7f24574f5780 1 < canceled part key=_multipart_SUBILSL92D96.2~ZBLB0QVIAOXA.1 size=219780 2023-12-03T11:48:41.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.245+0000 7f24574f5780 1 < canceled multipart upload key=SUBILSL92D96 upload=ZBLB0QVIAOXA tag=DLVHFZ289UVV type=write size=2316932 2023-12-03T11:48:41.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.247+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~G6D2Z7TBYURI.0 size=639708 2023-12-03T11:48:41.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.247+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=G6D2Z7TBYURI tag=G9N1RAJZ74JB type=write size=1688284 2023-12-03T11:48:41.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.248+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=2S1M54GII6C9 type=del 2023-12-03T11:48:41.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.250+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~CTQLZDZSGVL0.0 size=935675 2023-12-03T11:48:41.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.250+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=CTQLZDZSGVL0 tag=O0E5LCYBLHUN type=write size=1984251 2023-12-03T11:48:41.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.252+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.0 size=1048576 2023-12-03T11:48:41.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.252+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.1 size=1048576 2023-12-03T11:48:41.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.252+0000 7f24574f5780 1 < canceled part key=_multipart_A4LOQE7VRDN7.2~3O32ST39SO0H.2 size=974493 2023-12-03T11:48:41.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.254+0000 7f24574f5780 1 < canceled multipart upload key=A4LOQE7VRDN7 upload=3O32ST39SO0H tag=C2R7LXC56TKU type=write size=4120221 2023-12-03T11:48:41.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.256+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=ZEKZI827Y8JD type=del size=2372295 2023-12-03T11:48:41.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.257+0000 7f24574f5780 1 < completed write operation key=XO42YZR60KQE tag=I0XTUSBRGHKU type=write size=154478 2023-12-03T11:48:41.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.258+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=GJ8CM5WQRMHH type=del size=3502340 2023-12-03T11:48:41.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.259+0000 7f24574f5780 1 < completed write operation key=9LKFY1U0JRBQ tag=K5JP3IV6RHJD type=write size=412264 2023-12-03T11:48:41.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.261+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~OBN5DRTN14HX.0 size=425692 2023-12-03T11:48:41.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.261+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=OBN5DRTN14HX tag=K8QD9M9L5CBS type=write size=1474268 2023-12-03T11:48:41.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.262+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=PQAPT47Y7A5V type=write size=612044 2023-12-03T11:48:41.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.263+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=8TAHHPXDYTAI type=del size=1955425 2023-12-03T11:48:41.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.266+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=5JLDTF1BKJ89 type=del 2023-12-03T11:48:41.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.267+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=E105PKV75HFF type=del size=439883 2023-12-03T11:48:41.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.269+0000 7f24574f5780 1 < suggested remove operation key=XRHVYVADRWET tag=VXXMQFIHZ1MY type=del 2023-12-03T11:48:41.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.270+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=JTSPHYCXUEAG type=del size=4013712 2023-12-03T11:48:41.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.271+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~2SAT23YY4V15.0 size=829087 2023-12-03T11:48:41.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.274+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=2SAT23YY4V15 tag=U9VPQ3IO5JYO type=write size=1877663 2023-12-03T11:48:41.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.275+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=R2HWHU42UM4O type=write size=141814 2023-12-03T11:48:41.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.277+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=X1U93LLXQKWG type=del size=2571178 2023-12-03T11:48:41.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.279+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~6I5DON39GPXR.0 size=39401 2023-12-03T11:48:41.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.279+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=6I5DON39GPXR tag=8W8RLG3I5ZQ6 type=write size=1087977 2023-12-03T11:48:41.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.281+0000 7f24574f5780 1 < suggested update operation key=ON2E6H35N9FJ tag=3GI3X3N9WRLW type=del size=2885508 2023-12-03T11:48:41.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.283+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=TU8FQ8LOVX9C type=del size=1131775 2023-12-03T11:48:41.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.285+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=W1OMOM8UP1OP type=del 2023-12-03T11:48:41.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.287+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.0 size=1048576 2023-12-03T11:48:41.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.288+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.1 size=1048576 2023-12-03T11:48:41.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.289+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.2 size=963760 2023-12-03T11:48:41.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.289+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=02C55VYZ2YJS tag=TGQ0B661KG41 type=write size=4109488 2023-12-03T11:48:41.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.291+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=OTGGDCFZQNM9 type=del 2023-12-03T11:48:41.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.292+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=OCDV4CTYPF6Z type=write size=65694 2023-12-03T11:48:41.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.294+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=4CMSLRPIJGMT type=del 2023-12-03T11:48:41.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.296+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~U99E1XBY4U2C.0 size=129844 2023-12-03T11:48:41.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.296+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=U99E1XBY4U2C tag=ZCR6F1IXAO8D type=write size=1178420 2023-12-03T11:48:41.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.297+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=MQ9Z08223CZZ type=del 2023-12-03T11:48:41.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.299+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=Q2UG3UQOS2AJ type=del size=3515746 2023-12-03T11:48:41.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.300+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.0 size=1048576 2023-12-03T11:48:41.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.300+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.1 size=1048576 2023-12-03T11:48:41.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.301+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~L6FOM1XNZ0EA.2 size=1039669 2023-12-03T11:48:41.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.303+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=L6FOM1XNZ0EA tag=8XXS1XXTQILD type=write size=4185397 2023-12-03T11:48:41.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.305+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=G7WC256QH7NJ type=del size=2398202 2023-12-03T11:48:41.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.305+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~G6D2Z7TBYURI.0 size=639708 2023-12-03T11:48:41.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.307+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=G6D2Z7TBYURI tag=G9N1RAJZ74JB type=write size=1688284 2023-12-03T11:48:41.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.308+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=CN3RZRLTY40F type=del size=2136377 2023-12-03T11:48:41.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.310+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~CTQLZDZSGVL0.0 size=935675 2023-12-03T11:48:41.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.312+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=CTQLZDZSGVL0 tag=O0E5LCYBLHUN type=write size=1984251 2023-12-03T11:48:41.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.313+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=Z1BF941WSV1U type=del size=2287480 2023-12-03T11:48:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.315+0000 7f24574f5780 1 < suggested remove operation key=BV8C0VSJ1GTW tag=ZEKZI827Y8JD type=del 2023-12-03T11:48:41.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.316+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=OVTBIG2SWP0X type=del size=1936938 2023-12-03T11:48:41.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.317+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=GJ8CM5WQRMHH type=del 2023-12-03T11:48:41.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.319+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=G0WR3OLGZ1ZC type=del size=413138 2023-12-03T11:48:41.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.320+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~OBN5DRTN14HX.0 size=425692 2023-12-03T11:48:41.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.322+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=OBN5DRTN14HX tag=K8QD9M9L5CBS type=write size=1474268 2023-12-03T11:48:41.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.324+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~T8VWW0ELQ8MM.0 size=1048576 2023-12-03T11:48:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.325+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~T8VWW0ELQ8MM.1 size=53583 2023-12-03T11:48:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.325+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=T8VWW0ELQ8MM tag=0PMJR1JSL4HJ type=write size=2150735 2023-12-03T11:48:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.326+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=8TAHHPXDYTAI type=del 2023-12-03T11:48:41.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.327+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=XV1F1OY7TMM0 type=del size=55817 2023-12-03T11:48:41.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.329+0000 7f24574f5780 1 < suggested remove operation key=1V1C59DP8YPF tag=E105PKV75HFF type=del 2023-12-03T11:48:41.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.330+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=STJP9GCKCF0V type=del size=1154322 2023-12-03T11:48:41.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.332+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=JTSPHYCXUEAG type=del 2023-12-03T11:48:41.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.333+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=VD0OK5V2QWLU type=del size=309277 2023-12-03T11:48:41.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.334+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=R2HWHU42UM4O type=write size=141814 2023-12-03T11:48:41.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.335+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=GPWCBQNXX3UI type=del size=2719737 2023-12-03T11:48:41.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.336+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~6I5DON39GPXR.0 size=39401 2023-12-03T11:48:41.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.338+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=6I5DON39GPXR tag=8W8RLG3I5ZQ6 type=write size=1087977 2023-12-03T11:48:41.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.340+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.0 size=1048576 2023-12-03T11:48:41.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.341+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.1 size=1048576 2023-12-03T11:48:41.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.342+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.2 size=941568 2023-12-03T11:48:41.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.342+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=DLUEHDEIHRF1 tag=YN6USQ2Y6Q0V type=write size=4087296 2023-12-03T11:48:41.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.344+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=TU8FQ8LOVX9C type=del 2023-12-03T11:48:41.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.345+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=AFFPLLEH4FLF type=del size=1834199 2023-12-03T11:48:41.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.346+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.0 size=1048576 2023-12-03T11:48:41.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.347+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.1 size=1048576 2023-12-03T11:48:41.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.348+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~02C55VYZ2YJS.2 size=963760 2023-12-03T11:48:41.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.350+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=02C55VYZ2YJS tag=TGQ0B661KG41 type=write size=4109488 2023-12-03T11:48:41.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.351+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=DX8ZLUETQ3PX type=del size=2216390 2023-12-03T11:48:41.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.352+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=OCDV4CTYPF6Z type=write size=65694 2023-12-03T11:48:41.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.353+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=AO76K3HTX8LT type=del size=3438203 2023-12-03T11:48:41.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.355+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~U99E1XBY4U2C.0 size=129844 2023-12-03T11:48:41.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.357+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=U99E1XBY4U2C tag=ZCR6F1IXAO8D type=write size=1178420 2023-12-03T11:48:41.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.358+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=K44Q633BLGBE type=del size=548690 2023-12-03T11:48:41.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.359+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=Q2UG3UQOS2AJ type=del 2023-12-03T11:48:41.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.361+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.0 size=1048576 2023-12-03T11:48:41.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.362+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.1 size=1048576 2023-12-03T11:48:41.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.363+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.2 size=985189 2023-12-03T11:48:41.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.363+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=WJSHT18WWFRH tag=8OH2O1JWEUGL type=write size=4130917 2023-12-03T11:48:41.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.364+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=G7WC256QH7NJ type=del 2023-12-03T11:48:41.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.365+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=42SYRTRHD4YW type=write size=751475 2023-12-03T11:48:41.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.366+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=CN3RZRLTY40F type=del 2023-12-03T11:48:41.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.368+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~Q367KPD40H55.0 size=60594 2023-12-03T11:48:41.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.368+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=Q367KPD40H55 tag=5BEX285L5LY9 type=write size=1109170 2023-12-03T11:48:41.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.369+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=Z1BF941WSV1U type=del 2023-12-03T11:48:41.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.372+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FTNA4H5YQCAW.0 size=191212 2023-12-03T11:48:41.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.372+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=FTNA4H5YQCAW tag=HF20O4XVHWS9 type=write size=1239788 2023-12-03T11:48:41.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.374+0000 7f24574f5780 1 < suggested remove operation key=6WY91DFIRUN8 tag=OVTBIG2SWP0X type=del 2023-12-03T11:48:41.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.375+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=FCCQ3S9AX7AH type=del size=2148790 2023-12-03T11:48:41.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.376+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=G0WR3OLGZ1ZC type=del 2023-12-03T11:48:41.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.377+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=C55CXA9ZQGEK type=write size=224945 2023-12-03T11:48:41.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.378+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~T8VWW0ELQ8MM.0 size=1048576 2023-12-03T11:48:41.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.379+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~T8VWW0ELQ8MM.1 size=53583 2023-12-03T11:48:41.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.381+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=T8VWW0ELQ8MM tag=0PMJR1JSL4HJ type=write size=2150735 2023-12-03T11:48:41.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.383+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=QOPBH9M0JNR9 type=del size=6520 2023-12-03T11:48:41.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.384+0000 7f24574f5780 1 < canceled operation key=SC8UOVLQQ0VD tag=XV1F1OY7TMM0 type=del size=55817 2023-12-03T11:48:41.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.385+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=IM3QXM18Y680 type=del size=2898392 2023-12-03T11:48:41.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.386+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=STJP9GCKCF0V type=del 2023-12-03T11:48:41.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.387+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=EC2SMLLYDDON type=write size=187861 2023-12-03T11:48:41.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.388+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=VD0OK5V2QWLU type=del 2023-12-03T11:48:41.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.389+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=9HJT7YV134NP type=write size=1010280 2023-12-03T11:48:41.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.390+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=GPWCBQNXX3UI type=del 2023-12-03T11:48:41.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.392+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=0I99FKNZGX5N type=del size=4037929 2023-12-03T11:48:41.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.393+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.0 size=1048576 2023-12-03T11:48:41.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.394+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.1 size=1048576 2023-12-03T11:48:41.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.395+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~DLUEHDEIHRF1.2 size=941568 2023-12-03T11:48:41.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.397+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=DLUEHDEIHRF1 tag=YN6USQ2Y6Q0V type=write size=4087296 2023-12-03T11:48:41.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.398+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=D0DNXMBKIN5L type=del size=841643 2023-12-03T11:48:41.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.399+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=AFFPLLEH4FLF type=del 2023-12-03T11:48:41.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.401+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=FG63AJVFT4FU type=del size=1699918 2023-12-03T11:48:41.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.403+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=DX8ZLUETQ3PX type=del 2023-12-03T11:48:41.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.404+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=09N5SX0YYKL8 type=del size=1750799 2023-12-03T11:48:41.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.405+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=AO76K3HTX8LT type=del 2023-12-03T11:48:41.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.406+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=FXEB2F551I8C type=del size=178311 2023-12-03T11:48:41.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.408+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=K44Q633BLGBE type=del size=548690 2023-12-03T11:48:41.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.409+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=ZUAECXJRMOKZ type=del size=573894 2023-12-03T11:48:41.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.410+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.0 size=1048576 2023-12-03T11:48:41.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.411+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.1 size=1048576 2023-12-03T11:48:41.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.412+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~WJSHT18WWFRH.2 size=985189 2023-12-03T11:48:41.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.414+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=WJSHT18WWFRH tag=8OH2O1JWEUGL type=write size=4130917 2023-12-03T11:48:41.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.415+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=PYFUTM2U1BL4 type=del size=3876104 2023-12-03T11:48:41.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.416+0000 7f24574f5780 1 < completed write operation key=U043YAPGUBT5 tag=42SYRTRHD4YW type=write size=751475 2023-12-03T11:48:41.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.418+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~8E39660MFR65.0 size=935233 2023-12-03T11:48:41.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.418+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=8E39660MFR65 tag=YUTLKTTM0OAW type=write size=1983809 2023-12-03T11:48:41.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.419+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~Q367KPD40H55.0 size=60594 2023-12-03T11:48:41.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.421+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=Q367KPD40H55 tag=5BEX285L5LY9 type=write size=1109170 2023-12-03T11:48:41.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.423+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=KHO7WQMPQANF type=write size=741809 2023-12-03T11:48:41.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.424+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FTNA4H5YQCAW.0 size=191212 2023-12-03T11:48:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.426+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=FTNA4H5YQCAW tag=HF20O4XVHWS9 type=write size=1239788 2023-12-03T11:48:41.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.428+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.0 size=1048576 2023-12-03T11:48:41.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.429+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.1 size=1048576 2023-12-03T11:48:41.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.430+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.2 size=515516 2023-12-03T11:48:41.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.430+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=KOV1UT9RTN6J tag=FWUTHL5XPADF type=write size=3661244 2023-12-03T11:48:41.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.432+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=FCCQ3S9AX7AH type=del 2023-12-03T11:48:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.433+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=UZILLIRMDRTB type=del size=3514553 2023-12-03T11:48:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.434+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=C55CXA9ZQGEK type=write size=224945 2023-12-03T11:48:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.435+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=JI3OV7EY29QQ type=del size=2351021 2023-12-03T11:48:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.436+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=QOPBH9M0JNR9 type=del 2023-12-03T11:48:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.437+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=OWXSN0LTCY32 type=del size=3294492 2023-12-03T11:48:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.438+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=IM3QXM18Y680 type=del 2023-12-03T11:48:41.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.440+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~75AT2ZW1LBXK.0 size=1048576 2023-12-03T11:48:41.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.441+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~75AT2ZW1LBXK.1 size=508942 2023-12-03T11:48:41.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.441+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=75AT2ZW1LBXK tag=O402HZVHO295 type=write size=2606094 2023-12-03T11:48:41.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.443+0000 7f24574f5780 1 < completed write operation key=PGTGVLK4DBN0 tag=EC2SMLLYDDON type=write size=187861 2023-12-03T11:48:41.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.445+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~0428XB2G03TH.0 size=60441 2023-12-03T11:48:41.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.445+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=0428XB2G03TH tag=4PZTR1NO4LZR type=write size=1109017 2023-12-03T11:48:41.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.447+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=9HJT7YV134NP type=write 2023-12-03T11:48:41.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.449+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.0 size=1048576 2023-12-03T11:48:41.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.451+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.1 size=1048576 2023-12-03T11:48:41.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.452+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.2 size=891506 2023-12-03T11:48:41.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.452+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=PLOXINYJIUYX tag=B745U8DFFHLF type=write size=4037234 2023-12-03T11:48:41.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.453+0000 7f24574f5780 1 < canceled operation key=WD7U31FW3NGZ tag=0I99FKNZGX5N type=del size=4037929 2023-12-03T11:48:41.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.454+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=VLUPOXNV8HZC type=del size=420207 2023-12-03T11:48:41.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.455+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=D0DNXMBKIN5L type=del 2023-12-03T11:48:41.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.456+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=1F9FOWEJZ9JB type=write size=566306 2023-12-03T11:48:41.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.458+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=FG63AJVFT4FU type=del 2023-12-03T11:48:41.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.459+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=XVLJ2L0XF7RM type=del size=3421712 2023-12-03T11:48:41.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.460+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=09N5SX0YYKL8 type=del 2023-12-03T11:48:41.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.461+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=MVD3POLMWJC1 type=del size=292563 2023-12-03T11:48:41.464 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.462+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=FXEB2F551I8C type=del size=178311 2023-12-03T11:48:41.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.464+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.0 size=1048576 2023-12-03T11:48:41.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.465+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.1 size=1048576 2023-12-03T11:48:41.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.466+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.2 size=258531 2023-12-03T11:48:41.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.466+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=KSTLB98N2SJ9 tag=Z4NEJXUTCVBN type=write size=3404259 2023-12-03T11:48:41.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.467+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=ZUAECXJRMOKZ type=del 2023-12-03T11:48:41.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.469+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=B79J3W9RIN1V type=del size=1566669 2023-12-03T11:48:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.470+0000 7f24574f5780 -1 < failed to complete operation key=S73M6UFMX5NH tag=PYFUTM2U1BL4 type=del size=3876104: (22) Invalid argument 2023-12-03T11:48:41.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.471+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=W26MLBTNRKMN type=del size=2341029 2023-12-03T11:48:41.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.472+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~8E39660MFR65.0 size=935233 2023-12-03T11:48:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.474+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=8E39660MFR65 tag=YUTLKTTM0OAW type=write size=1983809 2023-12-03T11:48:41.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.475+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=8L1N16YTWOQV type=del size=3598430 2023-12-03T11:48:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.477+0000 7f24574f5780 1 < suggested update operation key=6WY91DFIRUN8 tag=KHO7WQMPQANF type=write size=741809 2023-12-03T11:48:41.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.480+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~P3GS0TOJQU90.0 size=1048576 2023-12-03T11:48:41.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.480+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~P3GS0TOJQU90.1 size=590139 2023-12-03T11:48:41.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.480+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=P3GS0TOJQU90 tag=8HOS2MXAYOZT type=write size=2687291 2023-12-03T11:48:41.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.482+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.0 size=1048576 2023-12-03T11:48:41.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.483+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.1 size=1048576 2023-12-03T11:48:41.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.483+0000 7f24574f5780 1 < canceled part key=_multipart_WZMWCSAZX8RB.2~KOV1UT9RTN6J.2 size=515516 2023-12-03T11:48:41.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.485+0000 7f24574f5780 1 < canceled multipart upload key=WZMWCSAZX8RB upload=KOV1UT9RTN6J tag=FWUTHL5XPADF type=write size=3661244 2023-12-03T11:48:41.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.486+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=WGQALH7X56LD type=del size=1480970 2023-12-03T11:48:41.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.487+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=UZILLIRMDRTB type=del 2023-12-03T11:48:41.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.489+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~189863GGRTXX.0 size=1048576 2023-12-03T11:48:41.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.490+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~189863GGRTXX.1 size=1036104 2023-12-03T11:48:41.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.490+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=189863GGRTXX tag=KFDJ58IXN8GE type=write size=3133256 2023-12-03T11:48:41.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.491+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=JI3OV7EY29QQ type=del 2023-12-03T11:48:41.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.492+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=72NW4E6V9T8V type=del size=4063124 2023-12-03T11:48:41.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.493+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=OWXSN0LTCY32 type=del 2023-12-03T11:48:41.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.495+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=YBWFJHZ0DS2R type=del size=2837538 2023-12-03T11:48:41.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.495+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~75AT2ZW1LBXK.0 size=1048576 2023-12-03T11:48:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.496+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~75AT2ZW1LBXK.1 size=508942 2023-12-03T11:48:41.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.498+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=75AT2ZW1LBXK tag=O402HZVHO295 type=write size=2606094 2023-12-03T11:48:41.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.499+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=YDIY2BX3H2UH type=write size=552532 2023-12-03T11:48:41.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.500+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~0428XB2G03TH.0 size=60441 2023-12-03T11:48:41.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.503+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=0428XB2G03TH tag=4PZTR1NO4LZR type=write size=1109017 2023-12-03T11:48:41.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.504+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=X9F82TKI0SIT type=write size=872456 2023-12-03T11:48:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.506+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.0 size=1048576 2023-12-03T11:48:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.507+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.1 size=1048576 2023-12-03T11:48:41.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.508+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PLOXINYJIUYX.2 size=891506 2023-12-03T11:48:41.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.510+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=PLOXINYJIUYX tag=B745U8DFFHLF type=write size=4037234 2023-12-03T11:48:41.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.512+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~SRO8PTH5QSLF.0 size=304744 2023-12-03T11:48:41.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.512+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=SRO8PTH5QSLF tag=4IVAQ0XL065M type=write size=1353320 2023-12-03T11:48:41.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.515+0000 7f24574f5780 1 < suggested remove operation key=RABUR02D2ZMD tag=VLUPOXNV8HZC type=del 2023-12-03T11:48:41.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.516+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=3YL6GQIZY4D1 type=write size=936485 2023-12-03T11:48:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.518+0000 7f24574f5780 1 < suggested remove operation key=PGTGVLK4DBN0 tag=1F9FOWEJZ9JB type=write 2023-12-03T11:48:41.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.520+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=PSPFUG6CAYO1 type=del size=1233288 2023-12-03T11:48:41.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.522+0000 7f24574f5780 1 < suggested remove operation key=XZFHFS76WNR4 tag=XVLJ2L0XF7RM type=del 2023-12-03T11:48:41.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.523+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=LMZ0JCN8399E type=write size=472356 2023-12-03T11:48:41.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.524+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=MVD3POLMWJC1 type=del size=292563 2023-12-03T11:48:41.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.526+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.0 size=1048576 2023-12-03T11:48:41.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.527+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.1 size=1048576 2023-12-03T11:48:41.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.528+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.2 size=653090 2023-12-03T11:48:41.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.528+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=IBQ7UAH1E6K8 tag=FC2MYJBESTNF type=write size=3798818 2023-12-03T11:48:41.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.529+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.0 size=1048576 2023-12-03T11:48:41.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.530+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.1 size=1048576 2023-12-03T11:48:41.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.530+0000 7f24574f5780 1 < canceled part key=_multipart_WD7U31FW3NGZ.2~KSTLB98N2SJ9.2 size=258531 2023-12-03T11:48:41.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.532+0000 7f24574f5780 1 < canceled multipart upload key=WD7U31FW3NGZ upload=KSTLB98N2SJ9 tag=Z4NEJXUTCVBN type=write size=3404259 2023-12-03T11:48:41.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.534+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~Y4B4FHEX9GEV.0 size=100238 2023-12-03T11:48:41.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.534+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=Y4B4FHEX9GEV tag=9NSP6N1XGVQG type=write size=1148814 2023-12-03T11:48:41.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.535+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=B79J3W9RIN1V type=del 2023-12-03T11:48:41.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.536+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=010ZTDIHOW1K type=del size=1099465 2023-12-03T11:48:41.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.537+0000 7f24574f5780 1 < canceled operation key=S73M6UFMX5NH tag=W26MLBTNRKMN type=del size=2341029 2023-12-03T11:48:41.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.539+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=GWER0UBNSVV7 type=write size=113593 2023-12-03T11:48:41.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.541+0000 7f24574f5780 1 < suggested remove operation key=1V39QA10M1HY tag=8L1N16YTWOQV type=del 2023-12-03T11:48:41.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.544+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~0Y0TIKJVRGIQ.0 size=1048576 2023-12-03T11:48:41.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.545+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~0Y0TIKJVRGIQ.1 size=336252 2023-12-03T11:48:41.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.545+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=0Y0TIKJVRGIQ tag=9Q0FL6ECQ418 type=write size=2433404 2023-12-03T11:48:41.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.546+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~P3GS0TOJQU90.0 size=1048576 2023-12-03T11:48:41.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.547+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~P3GS0TOJQU90.1 size=590139 2023-12-03T11:48:41.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.549+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=P3GS0TOJQU90 tag=8HOS2MXAYOZT type=write size=2687291 2023-12-03T11:48:41.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.550+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=142RO9HXLM75 type=write size=181832 2023-12-03T11:48:41.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.553+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=WGQALH7X56LD type=del size=1480970 2023-12-03T11:48:41.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.554+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=8FG1HK6NAHVZ type=del size=3171931 2023-12-03T11:48:41.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.555+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~189863GGRTXX.0 size=1048576 2023-12-03T11:48:41.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.556+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~189863GGRTXX.1 size=1036104 2023-12-03T11:48:41.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.558+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=189863GGRTXX tag=KFDJ58IXN8GE type=write size=3133256 2023-12-03T11:48:41.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.559+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=U97EAIKRTXLS type=del size=1678927 2023-12-03T11:48:41.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.560+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=72NW4E6V9T8V type=del 2023-12-03T11:48:41.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.562+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~XZT1JC0R2Y65.0 size=157547 2023-12-03T11:48:41.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.562+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=XZT1JC0R2Y65 tag=0I8BX08UHVM0 type=write size=1206123 2023-12-03T11:48:41.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.563+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=YBWFJHZ0DS2R type=del 2023-12-03T11:48:41.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.564+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=47VOLRFRIB4L type=del size=2177139 2023-12-03T11:48:41.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.566+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=YDIY2BX3H2UH type=write size=552532: (22) Invalid argument 2023-12-03T11:48:41.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.567+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=L3I003MIWEUG type=del size=4016792 2023-12-03T11:48:41.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.570+0000 7f24574f5780 1 < suggested remove operation key=SC8UOVLQQ0VD tag=X9F82TKI0SIT type=write 2023-12-03T11:48:41.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.572+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=71SM182BIUJA type=del size=2697083 2023-12-03T11:48:41.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.573+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~SRO8PTH5QSLF.0 size=304744 2023-12-03T11:48:41.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.575+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=SRO8PTH5QSLF tag=4IVAQ0XL065M type=write size=1353320 2023-12-03T11:48:41.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.577+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~AML3EG91XOF3.0 size=1048576 2023-12-03T11:48:41.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.578+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~AML3EG91XOF3.1 size=934151 2023-12-03T11:48:41.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.578+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=AML3EG91XOF3 tag=M212W7M01S4T type=write size=3031303 2023-12-03T11:48:41.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.579+0000 7f24574f5780 1 < canceled operation key=S73M6UFMX5NH tag=3YL6GQIZY4D1 type=write size=936485 2023-12-03T11:48:41.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.581+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~F23E0SU7TY29.0 size=1048576 2023-12-03T11:48:41.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.582+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~F23E0SU7TY29.1 size=506899 2023-12-03T11:48:41.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.582+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=F23E0SU7TY29 tag=PFQ3WD4PGG66 type=write size=2604051 2023-12-03T11:48:41.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.584+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=PSPFUG6CAYO1 type=del 2023-12-03T11:48:41.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.586+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.0 size=1048576 2023-12-03T11:48:41.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.587+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.1 size=1048576 2023-12-03T11:48:41.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.588+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.2 size=1028855 2023-12-03T11:48:41.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.588+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=B3Q7XY8C9QTC tag=KBV7OY6GGIMJ type=write size=4174583 2023-12-03T11:48:41.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.589+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=LMZ0JCN8399E type=write size=472356 2023-12-03T11:48:41.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.590+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=MC888W7V1NEP type=del size=3134627 2023-12-03T11:48:41.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.592+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.0 size=1048576 2023-12-03T11:48:41.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.593+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.1 size=1048576 2023-12-03T11:48:41.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.593+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~IBQ7UAH1E6K8.2 size=653090 2023-12-03T11:48:41.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.595+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=IBQ7UAH1E6K8 tag=FC2MYJBESTNF type=write size=3798818 2023-12-03T11:48:41.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.596+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=2SSWW8SRSZV3 type=del size=2132618 2023-12-03T11:48:41.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.597+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~Y4B4FHEX9GEV.0 size=100238 2023-12-03T11:48:41.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.599+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=Y4B4FHEX9GEV tag=9NSP6N1XGVQG type=write size=1148814 2023-12-03T11:48:41.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.600+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=MDVDD1O8L2C8 type=write size=640011 2023-12-03T11:48:41.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.601+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=010ZTDIHOW1K type=del 2023-12-03T11:48:41.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.603+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.0 size=1048576 2023-12-03T11:48:41.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.604+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.1 size=1048576 2023-12-03T11:48:41.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.605+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.2 size=127717 2023-12-03T11:48:41.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.605+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=22BRVDEY8D3Z tag=4Q5HH2N2F837 type=write size=3273445 2023-12-03T11:48:41.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.606+0000 7f24574f5780 1 < completed write operation key=XRHVYVADRWET tag=GWER0UBNSVV7 type=write size=113593 2023-12-03T11:48:41.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.608+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.0 size=1048576 2023-12-03T11:48:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.609+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.1 size=1048576 2023-12-03T11:48:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.610+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.2 size=832177 2023-12-03T11:48:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.610+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=G32UYTXXVH7V tag=NA09JBCGKWYN type=write size=3977905 2023-12-03T11:48:41.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.611+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~0Y0TIKJVRGIQ.0 size=1048576 2023-12-03T11:48:41.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.612+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~0Y0TIKJVRGIQ.1 size=336252 2023-12-03T11:48:41.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.614+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=0Y0TIKJVRGIQ tag=9Q0FL6ECQ418 type=write size=2433404 2023-12-03T11:48:41.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.615+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=010LNVMH23EE type=del size=2411292 2023-12-03T11:48:41.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.616+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=142RO9HXLM75 type=write size=181832 2023-12-03T11:48:41.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.618+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~13X7TRXKHD72.0 size=1048576 2023-12-03T11:48:41.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.619+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~13X7TRXKHD72.1 size=842267 2023-12-03T11:48:41.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.619+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=13X7TRXKHD72 tag=ZH0AVXT7K0NJ type=write size=2939419 2023-12-03T11:48:41.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.620+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=8FG1HK6NAHVZ type=del 2023-12-03T11:48:41.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.622+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1B6H766JZ2IU.0 size=272313 2023-12-03T11:48:41.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.622+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=1B6H766JZ2IU tag=XCEDQUME3XCF type=write size=1320889 2023-12-03T11:48:41.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.625+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=U97EAIKRTXLS type=del 2023-12-03T11:48:41.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.626+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=PDU6YO7V9REQ type=write size=391165 2023-12-03T11:48:41.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.627+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~XZT1JC0R2Y65.0 size=157547 2023-12-03T11:48:41.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.629+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=XZT1JC0R2Y65 tag=0I8BX08UHVM0 type=write size=1206123 2023-12-03T11:48:41.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.630+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=YMHMKXU0V0AP type=del size=929707 2023-12-03T11:48:41.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.633+0000 7f24574f5780 1 < suggested update operation key=HM9JHOV8W24F tag=47VOLRFRIB4L type=del size=2177139 2023-12-03T11:48:41.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.634+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=SIEC2UNA2O2I type=del size=1283353 2023-12-03T11:48:41.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.635+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=L3I003MIWEUG type=del 2023-12-03T11:48:41.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.638+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~3U3390NC9UQF.0 size=1048576 2023-12-03T11:48:41.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.638+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~3U3390NC9UQF.1 size=663700 2023-12-03T11:48:41.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.638+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=3U3390NC9UQF tag=C7GF8G67UHIM type=write size=2760852 2023-12-03T11:48:41.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.640+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=71SM182BIUJA type=del 2023-12-03T11:48:41.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.642+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.0 size=1048576 2023-12-03T11:48:41.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.643+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.1 size=1048576 2023-12-03T11:48:41.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.643+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.2 size=499574 2023-12-03T11:48:41.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.643+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=5NRY5HZ2J03X tag=UMD0W1XO76KE type=write size=3645302 2023-12-03T11:48:41.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.645+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~AML3EG91XOF3.0 size=1048576 2023-12-03T11:48:41.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.646+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~AML3EG91XOF3.1 size=934151 2023-12-03T11:48:41.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.648+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=AML3EG91XOF3 tag=M212W7M01S4T type=write size=3031303 2023-12-03T11:48:41.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.649+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=05E89Z16INYP type=del size=919028 2023-12-03T11:48:41.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.650+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~F23E0SU7TY29.0 size=1048576 2023-12-03T11:48:41.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.651+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~F23E0SU7TY29.1 size=506899 2023-12-03T11:48:41.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.653+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=F23E0SU7TY29 tag=PFQ3WD4PGG66 type=write size=2604051 2023-12-03T11:48:41.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.654+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=LCPLWMKL7CTH type=del size=1133337 2023-12-03T11:48:41.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.655+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.0 size=1048576 2023-12-03T11:48:41.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.656+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.1 size=1048576 2023-12-03T11:48:41.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.657+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~B3Q7XY8C9QTC.2 size=1028855 2023-12-03T11:48:41.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.659+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=B3Q7XY8C9QTC tag=KBV7OY6GGIMJ type=write size=4174583 2023-12-03T11:48:41.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.661+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~T3WGAF5QY7GD.0 size=506422 2023-12-03T11:48:41.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.661+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=T3WGAF5QY7GD tag=PVV3ZOW182PM type=write size=1554998 2023-12-03T11:48:41.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.662+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=MC888W7V1NEP type=del 2023-12-03T11:48:41.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.664+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~0D3RS76Q15KN.0 size=328133 2023-12-03T11:48:41.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.664+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=0D3RS76Q15KN tag=V503SGVEN5VX type=write size=1376709 2023-12-03T11:48:41.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.666+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=2SSWW8SRSZV3 type=del 2023-12-03T11:48:41.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.668+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~YDDX0FWS1A26.0 size=580464 2023-12-03T11:48:41.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.668+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=YDDX0FWS1A26 tag=L1BELX4TXIXQ type=write size=1629040 2023-12-03T11:48:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.669+0000 7f24574f5780 1 < completed write operation key=WD7U31FW3NGZ tag=MDVDD1O8L2C8 type=write size=640011 2023-12-03T11:48:41.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.671+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.0 size=1048576 2023-12-03T11:48:41.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.672+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.1 size=1048576 2023-12-03T11:48:41.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.673+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.2 size=91885 2023-12-03T11:48:41.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.673+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=FHJ1UIBYIBDE tag=NO4XG0VDA14W type=write size=3237613 2023-12-03T11:48:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.674+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.0 size=1048576 2023-12-03T11:48:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.675+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.1 size=1048576 2023-12-03T11:48:41.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.676+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~22BRVDEY8D3Z.2 size=127717 2023-12-03T11:48:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.678+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=22BRVDEY8D3Z tag=4Q5HH2N2F837 type=write size=3273445 2023-12-03T11:48:41.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.679+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=MCA5QWV7FI2D type=del size=548133 2023-12-03T11:48:41.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.680+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.0 size=1048576 2023-12-03T11:48:41.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.681+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.1 size=1048576 2023-12-03T11:48:41.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.682+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~G32UYTXXVH7V.2 size=832177 2023-12-03T11:48:41.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.683+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=G32UYTXXVH7V tag=NA09JBCGKWYN type=write size=3977905 2023-12-03T11:48:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.687+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~TNUCAD14JEQF.0 size=684819 2023-12-03T11:48:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.687+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=TNUCAD14JEQF tag=8SU6OEILR1PB type=write size=1733395 2023-12-03T11:48:41.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.688+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=010LNVMH23EE type=del 2023-12-03T11:48:41.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.689+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=QZ9BFWB3VFCA type=write size=523553 2023-12-03T11:48:41.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.690+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~13X7TRXKHD72.0 size=1048576 2023-12-03T11:48:41.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.690+0000 7f24574f5780 1 < canceled part key=_multipart_QVLPCWC4T8N8.2~13X7TRXKHD72.1 size=842267 2023-12-03T11:48:41.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.692+0000 7f24574f5780 1 < canceled multipart upload key=QVLPCWC4T8N8 upload=13X7TRXKHD72 tag=ZH0AVXT7K0NJ type=write size=2939419 2023-12-03T11:48:41.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.693+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=WUCW1LTAFCSU type=del size=3302932 2023-12-03T11:48:41.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.694+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1B6H766JZ2IU.0 size=272313 2023-12-03T11:48:41.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.696+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=1B6H766JZ2IU tag=XCEDQUME3XCF type=write size=1320889 2023-12-03T11:48:41.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.697+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=VAAFW3F35V0K type=del size=309101 2023-12-03T11:48:41.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.700+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=PDU6YO7V9REQ type=write size=391165 2023-12-03T11:48:41.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.701+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=PILKTCHV1O4W type=del size=3031398 2023-12-03T11:48:41.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.702+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=YMHMKXU0V0AP type=del 2023-12-03T11:48:41.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.703+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=O387VTEHHGRC type=del size=2721991 2023-12-03T11:48:41.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.704+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=SIEC2UNA2O2I type=del 2023-12-03T11:48:41.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.705+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=7CR16GF0WPN1 type=write size=314151 2023-12-03T11:48:41.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.707+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~3U3390NC9UQF.0 size=1048576 2023-12-03T11:48:41.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.708+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~3U3390NC9UQF.1 size=663700 2023-12-03T11:48:41.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.710+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=3U3390NC9UQF tag=C7GF8G67UHIM type=write size=2760852 2023-12-03T11:48:41.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.712+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~2Y3PEO6GVJ8X.0 size=1048576 2023-12-03T11:48:41.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.712+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~2Y3PEO6GVJ8X.1 size=419741 2023-12-03T11:48:41.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.713+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=2Y3PEO6GVJ8X tag=NOWSV31S25VR type=write size=2516893 2023-12-03T11:48:41.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.714+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.0 size=1048576 2023-12-03T11:48:41.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.715+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.1 size=1048576 2023-12-03T11:48:41.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.716+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~5NRY5HZ2J03X.2 size=499574 2023-12-03T11:48:41.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.717+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=5NRY5HZ2J03X tag=UMD0W1XO76KE type=write size=3645302 2023-12-03T11:48:41.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.719+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=74BT6Q9Q77NW type=del size=4099547 2023-12-03T11:48:41.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.721+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=05E89Z16INYP type=del 2023-12-03T11:48:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.722+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=O8Y7CM9JRSKC type=del size=880591 2023-12-03T11:48:41.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.723+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=LCPLWMKL7CTH type=del 2023-12-03T11:48:41.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.724+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=ILXDOHW6YZZ0 type=del size=3476211 2023-12-03T11:48:41.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.725+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~T3WGAF5QY7GD.0 size=506422 2023-12-03T11:48:41.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.727+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=T3WGAF5QY7GD tag=PVV3ZOW182PM type=write size=1554998 2023-12-03T11:48:41.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.728+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=GZGZSUZCHZCH type=del size=1863222 2023-12-03T11:48:41.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.729+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~0D3RS76Q15KN.0 size=328133 2023-12-03T11:48:41.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.732+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=0D3RS76Q15KN tag=V503SGVEN5VX type=write size=1376709 2023-12-03T11:48:41.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.734+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~IKZCICWTUDPA.0 size=1048576 2023-12-03T11:48:41.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.735+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~IKZCICWTUDPA.1 size=470938 2023-12-03T11:48:41.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.735+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=IKZCICWTUDPA tag=8FBTRKM6YV9A type=write size=2568090 2023-12-03T11:48:41.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.736+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~YDDX0FWS1A26.0 size=580464 2023-12-03T11:48:41.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.738+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=YDDX0FWS1A26 tag=L1BELX4TXIXQ type=write size=1629040 2023-12-03T11:48:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.739+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=97IZTB7IITMU type=del size=3900626 2023-12-03T11:48:41.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.740+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.0 size=1048576 2023-12-03T11:48:41.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.741+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.1 size=1048576 2023-12-03T11:48:41.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.742+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~FHJ1UIBYIBDE.2 size=91885 2023-12-03T11:48:41.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.744+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=FHJ1UIBYIBDE tag=NO4XG0VDA14W type=write size=3237613 2023-12-03T11:48:41.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.745+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=TQX6ZVBT1351 type=del size=3715011 2023-12-03T11:48:41.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.747+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=MCA5QWV7FI2D type=del 2023-12-03T11:48:41.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.748+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=O4GKP0409FEB type=del size=3544746 2023-12-03T11:48:41.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.749+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~TNUCAD14JEQF.0 size=684819 2023-12-03T11:48:41.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.751+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=TNUCAD14JEQF tag=8SU6OEILR1PB type=write size=1733395 2023-12-03T11:48:41.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.752+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=K5U10Y1Z05H1 type=del size=2057395 2023-12-03T11:48:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.753+0000 7f24574f5780 1 < canceled operation key=U043YAPGUBT5 tag=QZ9BFWB3VFCA type=write size=523553 2023-12-03T11:48:41.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.755+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~FQ1B3CHQMJYD.0 size=1048576 2023-12-03T11:48:41.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.756+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~FQ1B3CHQMJYD.1 size=756353 2023-12-03T11:48:41.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.756+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=FQ1B3CHQMJYD tag=IR9PTI1NQQCH type=write size=2853505 2023-12-03T11:48:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.757+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=WUCW1LTAFCSU type=del 2023-12-03T11:48:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.759+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2GQC8WP31KYD.0 size=867629 2023-12-03T11:48:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.759+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=2GQC8WP31KYD tag=0PB46UDAD2NK type=write size=1916205 2023-12-03T11:48:41.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.761+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=VAAFW3F35V0K type=del 2023-12-03T11:48:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.763+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.0 size=1048576 2023-12-03T11:48:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.764+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.1 size=1048576 2023-12-03T11:48:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.765+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.2 size=735034 2023-12-03T11:48:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.765+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=F0TBBRNG6CY8 tag=95EER2KHPOB0 type=write size=3880762 2023-12-03T11:48:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.767+0000 7f24574f5780 1 < suggested remove operation key=6WY91DFIRUN8 tag=PILKTCHV1O4W type=del 2023-12-03T11:48:41.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.769+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=0BIUO2358WZ6 type=del size=2893818 2023-12-03T11:48:41.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.771+0000 7f24574f5780 1 < suggested update operation key=HM9JHOV8W24F tag=O387VTEHHGRC type=del size=2721991 2023-12-03T11:48:41.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.773+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~JI3R2Q4WITIB.0 size=60436 2023-12-03T11:48:41.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.773+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=JI3R2Q4WITIB tag=6ZSZAW8UKWGN type=write size=1109012 2023-12-03T11:48:41.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.776+0000 7f24574f5780 1 < suggested remove operation key=1V1C59DP8YPF tag=7CR16GF0WPN1 type=write 2023-12-03T11:48:41.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.777+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=LWATKLHRHE0W type=write size=109054 2023-12-03T11:48:41.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.778+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~2Y3PEO6GVJ8X.0 size=1048576 2023-12-03T11:48:41.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.781+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~2Y3PEO6GVJ8X.1 size=419741 2023-12-03T11:48:41.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.783+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=2Y3PEO6GVJ8X tag=NOWSV31S25VR type=write size=2516893 2023-12-03T11:48:41.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.785+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~4YW4BS8H8SXY.0 size=1048576 2023-12-03T11:48:41.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.786+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~4YW4BS8H8SXY.1 size=353001 2023-12-03T11:48:41.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.786+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=4YW4BS8H8SXY tag=LOB9RN0BWIZ6 type=write size=2450153 2023-12-03T11:48:41.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.787+0000 7f24574f5780 -1 < failed to complete operation key=1V1C59DP8YPF tag=74BT6Q9Q77NW type=del size=4099547: (22) Invalid argument 2023-12-03T11:48:41.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.788+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=IN5A5VHQB2KC type=del size=1337083 2023-12-03T11:48:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.789+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=O8Y7CM9JRSKC type=del 2023-12-03T11:48:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.790+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=KL57AZS25ORI type=del size=1651285 2023-12-03T11:48:41.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.791+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=ILXDOHW6YZZ0 type=del 2023-12-03T11:48:41.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.793+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=JWCZMQ5W49Z9 type=del size=601670 2023-12-03T11:48:41.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.795+0000 7f24574f5780 1 < suggested remove operation key=BMKG19L35HLV tag=GZGZSUZCHZCH type=del 2023-12-03T11:48:41.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.796+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=3MN6111518NB type=del size=929927 2023-12-03T11:48:41.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.796+0000 7f24574f5780 1 < canceled part key=_multipart_XRHVYVADRWET.2~IKZCICWTUDPA.0 size=1048576 2023-12-03T11:48:41.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.797+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~IKZCICWTUDPA.1 size=470938 2023-12-03T11:48:41.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.799+0000 7f24574f5780 1 < canceled multipart upload key=XRHVYVADRWET upload=IKZCICWTUDPA tag=8FBTRKM6YV9A type=write size=2568090 2023-12-03T11:48:41.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.802+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=Q7XDE3122L01 type=del size=4056184 2023-12-03T11:48:41.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.803+0000 7f24574f5780 -1 < failed to complete operation key=HM9JHOV8W24F tag=97IZTB7IITMU type=del size=3900626: (22) Invalid argument 2023-12-03T11:48:41.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.804+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=6QUVXN5E3YXK type=del size=3754666 2023-12-03T11:48:41.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.806+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=TQX6ZVBT1351 type=del 2023-12-03T11:48:41.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.807+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=GKPUVEB3PDQN type=del size=2284545 2023-12-03T11:48:41.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.808+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=O4GKP0409FEB type=del 2023-12-03T11:48:41.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.809+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=HKGAMR53LW95 type=del size=3164545 2023-12-03T11:48:41.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.810+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=K5U10Y1Z05H1 type=del 2023-12-03T11:48:41.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.812+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=2UR9PJ0XJPW0 type=del size=1894900 2023-12-03T11:48:41.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.813+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~FQ1B3CHQMJYD.0 size=1048576 2023-12-03T11:48:41.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.814+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~FQ1B3CHQMJYD.1 size=756353 2023-12-03T11:48:41.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.816+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=FQ1B3CHQMJYD tag=IR9PTI1NQQCH type=write size=2853505 2023-12-03T11:48:41.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.817+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=VTF12K4EGRBJ type=del size=1936 2023-12-03T11:48:41.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.818+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2GQC8WP31KYD.0 size=867629 2023-12-03T11:48:41.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.820+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=2GQC8WP31KYD tag=0PB46UDAD2NK type=write size=1916205 2023-12-03T11:48:41.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.822+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.0 size=1048576 2023-12-03T11:48:41.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.823+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.1 size=1048576 2023-12-03T11:48:41.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.824+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.2 size=123794 2023-12-03T11:48:41.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.824+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=YZ2GLK4W7Z79 tag=0CAM93NAIP9U type=write size=3269522 2023-12-03T11:48:41.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.825+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.0 size=1048576 2023-12-03T11:48:41.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.826+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.1 size=1048576 2023-12-03T11:48:41.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.827+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~F0TBBRNG6CY8.2 size=735034 2023-12-03T11:48:41.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.829+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=F0TBBRNG6CY8 tag=95EER2KHPOB0 type=write size=3880762 2023-12-03T11:48:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.832+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~O9RCMLW9ITAO.0 size=960309 2023-12-03T11:48:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.832+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=O9RCMLW9ITAO tag=8OAQHTXBYUSO type=write size=2008885 2023-12-03T11:48:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.833+0000 7f24574f5780 1 < canceled operation key=PGTGVLK4DBN0 tag=0BIUO2358WZ6 type=del size=2893818 2023-12-03T11:48:41.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.834+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=CKP8Q8N76EDY type=write size=13147 2023-12-03T11:48:41.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.835+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~JI3R2Q4WITIB.0 size=60436 2023-12-03T11:48:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.837+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=JI3R2Q4WITIB tag=6ZSZAW8UKWGN type=write size=1109012 2023-12-03T11:48:41.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.840+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~S0H59C0UR8NO.0 size=1048576 2023-12-03T11:48:41.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.840+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~S0H59C0UR8NO.1 size=680250 2023-12-03T11:48:41.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.840+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=S0H59C0UR8NO tag=UAWE9SM7B05Y type=write size=2777402 2023-12-03T11:48:41.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.842+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=LWATKLHRHE0W type=write size=109054 2023-12-03T11:48:41.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.843+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=VRAP8NFXIZ5R type=write size=959986 2023-12-03T11:48:41.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.844+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~4YW4BS8H8SXY.0 size=1048576 2023-12-03T11:48:41.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.846+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~4YW4BS8H8SXY.1 size=353001 2023-12-03T11:48:41.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.847+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=4YW4BS8H8SXY tag=LOB9RN0BWIZ6 type=write size=2450153 2023-12-03T11:48:41.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.849+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=A2ZEVWEF3MAC type=del size=301291 2023-12-03T11:48:41.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.850+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=IN5A5VHQB2KC type=del 2023-12-03T11:48:41.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.851+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=8W0K4NFWV8EE type=del size=3569946 2023-12-03T11:48:41.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.853+0000 7f24574f5780 1 < suggested update operation key=1QR0TE9DXU5Z tag=KL57AZS25ORI type=del size=1651285 2023-12-03T11:48:41.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.854+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=3SPYKJ7VC1UT type=del size=1664774 2023-12-03T11:48:41.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.855+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=JWCZMQ5W49Z9 type=del 2023-12-03T11:48:41.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.858+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~6LHIPXPVY43M.0 size=1048576 2023-12-03T11:48:41.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.858+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~6LHIPXPVY43M.1 size=550015 2023-12-03T11:48:41.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.858+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=6LHIPXPVY43M tag=WLXNS5VS62VJ type=write size=2647167 2023-12-03T11:48:41.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.860+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=3MN6111518NB type=del 2023-12-03T11:48:41.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.861+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=9R87ACKRBIX6 type=del size=2250140 2023-12-03T11:48:41.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.862+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=Q7XDE3122L01 type=del 2023-12-03T11:48:41.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.864+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~HEM4JRV5CKGP.0 size=974555 2023-12-03T11:48:41.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.864+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=HEM4JRV5CKGP tag=S59PN3Q8F71H type=write size=2023131 2023-12-03T11:48:41.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.866+0000 7f24574f5780 1 < canceled operation key=MH1SRAATMNXB tag=6QUVXN5E3YXK type=del size=3754666 2023-12-03T11:48:41.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.868+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~Y9H8AWOEUHPP.0 size=252254 2023-12-03T11:48:41.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.868+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=Y9H8AWOEUHPP tag=2EUSN135YSAH type=write size=1300830 2023-12-03T11:48:41.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.869+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=GKPUVEB3PDQN type=del 2023-12-03T11:48:41.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.871+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.0 size=1048576 2023-12-03T11:48:41.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.872+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.1 size=1048576 2023-12-03T11:48:41.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.873+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.2 size=864649 2023-12-03T11:48:41.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.873+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=5DTFJ69XI1U3 tag=9DAFJYYSA7S6 type=write size=4010377 2023-12-03T11:48:41.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.874+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=HKGAMR53LW95 type=del 2023-12-03T11:48:41.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.875+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=705BZZL75RTG type=del size=814199 2023-12-03T11:48:41.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.877+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=2UR9PJ0XJPW0 type=del 2023-12-03T11:48:41.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.878+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=M2ML4QF7B39K type=del size=823008 2023-12-03T11:48:41.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.879+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=VTF12K4EGRBJ type=del 2023-12-03T11:48:41.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.881+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.0 size=1048576 2023-12-03T11:48:41.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.882+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.1 size=1048576 2023-12-03T11:48:41.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.883+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.2 size=381523 2023-12-03T11:48:41.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.883+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=0HLUPX1P0989 tag=TQ1OLRET248O type=write size=3527251 2023-12-03T11:48:41.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.885+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.0 size=1048576 2023-12-03T11:48:41.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.885+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.1 size=1048576 2023-12-03T11:48:41.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.886+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~YZ2GLK4W7Z79.2 size=123794 2023-12-03T11:48:41.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.888+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=YZ2GLK4W7Z79 tag=0CAM93NAIP9U type=write size=3269522 2023-12-03T11:48:41.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.890+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~NPY04SQ8LBDV.0 size=896158 2023-12-03T11:48:41.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.890+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=NPY04SQ8LBDV tag=A7ESCUIYJQTX type=write size=1944734 2023-12-03T11:48:41.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.892+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~O9RCMLW9ITAO.0 size=960309 2023-12-03T11:48:41.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.893+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=O9RCMLW9ITAO tag=8OAQHTXBYUSO type=write size=2008885 2023-12-03T11:48:41.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.895+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=0WMAXZJINWUV type=del size=1350388 2023-12-03T11:48:41.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.896+0000 7f24574f5780 1 < completed write operation key=1V39QA10M1HY tag=CKP8Q8N76EDY type=write size=13147 2023-12-03T11:48:41.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.897+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=F2A3N95C70FO type=del size=2027909 2023-12-03T11:48:41.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.898+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~S0H59C0UR8NO.0 size=1048576 2023-12-03T11:48:41.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.899+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~S0H59C0UR8NO.1 size=680250 2023-12-03T11:48:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.901+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=S0H59C0UR8NO tag=UAWE9SM7B05Y type=write size=2777402 2023-12-03T11:48:41.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.902+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=FRJYEV93SYG6 type=del size=3663784 2023-12-03T11:48:41.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.903+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=VRAP8NFXIZ5R type=write size=959986 2023-12-03T11:48:41.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.904+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=TJ921L3DOJ0W type=del size=862085 2023-12-03T11:48:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.905+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=A2ZEVWEF3MAC type=del 2023-12-03T11:48:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.907+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=ZXILUTU75G38 type=del size=2868521 2023-12-03T11:48:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.908+0000 7f24574f5780 1 < canceled operation key=MY3K9M4QRS3L tag=8W0K4NFWV8EE type=del size=3569946 2023-12-03T11:48:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.909+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=6T5X7VBQ2EV0 type=del size=1046485 2023-12-03T11:48:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.911+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=3SPYKJ7VC1UT type=del size=1664774 2023-12-03T11:48:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.914+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~Z621AQS66NUI.0 size=905808 2023-12-03T11:48:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.914+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=Z621AQS66NUI tag=HTYVMLDT7XWX type=write size=1954384 2023-12-03T11:48:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.915+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~6LHIPXPVY43M.0 size=1048576 2023-12-03T11:48:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.916+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~6LHIPXPVY43M.1 size=550015 2023-12-03T11:48:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.918+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=6LHIPXPVY43M tag=WLXNS5VS62VJ type=write size=2647167 2023-12-03T11:48:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.920+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KHODDFPHYXA8.0 size=1048576 2023-12-03T11:48:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.921+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KHODDFPHYXA8.1 size=528855 2023-12-03T11:48:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.921+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=KHODDFPHYXA8 tag=TFRRBONQGKW5 type=write size=2626007 2023-12-03T11:48:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.923+0000 7f24574f5780 1 < suggested update operation key=HM9JHOV8W24F tag=9R87ACKRBIX6 type=del size=2250140 2023-12-03T11:48:41.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.924+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=L8DLA5U7JE4F type=del size=402242 2023-12-03T11:48:41.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.925+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~HEM4JRV5CKGP.0 size=974555 2023-12-03T11:48:41.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.927+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=HEM4JRV5CKGP tag=S59PN3Q8F71H type=write size=2023131 2023-12-03T11:48:41.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.928+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=HYLEY960MC50 type=del size=2843137 2023-12-03T11:48:41.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.929+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~Y9H8AWOEUHPP.0 size=252254 2023-12-03T11:48:41.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.931+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=Y9H8AWOEUHPP tag=2EUSN135YSAH type=write size=1300830 2023-12-03T11:48:41.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.933+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~L145W8SXOKIX.0 size=483484 2023-12-03T11:48:41.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.933+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=L145W8SXOKIX tag=065Q8692NHNV type=write size=1532060 2023-12-03T11:48:41.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.935+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.0 size=1048576 2023-12-03T11:48:41.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.935+0000 7f24574f5780 1 < canceled part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.1 size=1048576 2023-12-03T11:48:41.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.936+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~5DTFJ69XI1U3.2 size=864649 2023-12-03T11:48:41.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.938+0000 7f24574f5780 1 < canceled multipart upload key=A4LOQE7VRDN7 upload=5DTFJ69XI1U3 tag=9DAFJYYSA7S6 type=write size=4010377 2023-12-03T11:48:41.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.940+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~NEYREFJOLJG9.0 size=1048576 2023-12-03T11:48:41.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.941+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~NEYREFJOLJG9.1 size=773382 2023-12-03T11:48:41.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.941+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=NEYREFJOLJG9 tag=EXBHB68XKFRL type=write size=2870534 2023-12-03T11:48:41.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.942+0000 7f24574f5780 1 < canceled operation key=6WY91DFIRUN8 tag=705BZZL75RTG type=del size=814199 2023-12-03T11:48:41.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.943+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=A84I9QON73BN type=del size=790952 2023-12-03T11:48:41.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.945+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=M2ML4QF7B39K type=del 2023-12-03T11:48:41.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.946+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=37EH1YM993KE type=del size=3883857 2023-12-03T11:48:41.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.947+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.0 size=1048576 2023-12-03T11:48:41.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.948+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.1 size=1048576 2023-12-03T11:48:41.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.949+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~0HLUPX1P0989.2 size=381523 2023-12-03T11:48:41.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.951+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=0HLUPX1P0989 tag=TQ1OLRET248O type=write size=3527251 2023-12-03T11:48:41.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.954+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~50VQ0DT96EO1.0 size=1048576 2023-12-03T11:48:41.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.955+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~50VQ0DT96EO1.1 size=627383 2023-12-03T11:48:41.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.955+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=50VQ0DT96EO1 tag=S39R75FW4S38 type=write size=2724535 2023-12-03T11:48:41.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.956+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~NPY04SQ8LBDV.0 size=896158 2023-12-03T11:48:41.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.958+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=NPY04SQ8LBDV tag=A7ESCUIYJQTX type=write size=1944734 2023-12-03T11:48:41.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.959+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=C8GPEB0V0HMT type=del size=4005006 2023-12-03T11:48:41.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.961+0000 7f24574f5780 1 < suggested update operation key=S0EH0Q0UA8HG tag=0WMAXZJINWUV type=del size=1350388 2023-12-03T11:48:41.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.962+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=J03UNNOGZRW0 type=del size=113154 2023-12-03T11:48:41.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.963+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=F2A3N95C70FO type=del 2023-12-03T11:48:41.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.965+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.0 size=1048576 2023-12-03T11:48:41.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.966+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.1 size=1048576 2023-12-03T11:48:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.967+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.2 size=205445 2023-12-03T11:48:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.967+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=LVOHPUSF1SYZ tag=4T2SB32X9UAZ type=write size=3351173 2023-12-03T11:48:41.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.969+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=FRJYEV93SYG6 type=del 2023-12-03T11:48:41.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.970+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=L4W51ZT2UIXH type=del size=3424768 2023-12-03T11:48:41.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.971+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=TJ921L3DOJ0W type=del 2023-12-03T11:48:41.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.972+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=PA1YANYO37EZ type=del size=1246890 2023-12-03T11:48:41.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.973+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=ZXILUTU75G38 type=del 2023-12-03T11:48:41.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.975+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=O7ZE3I7AWPD7 type=del size=466352 2023-12-03T11:48:41.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.976+0000 7f24574f5780 1 < canceled operation key=6WY91DFIRUN8 tag=6T5X7VBQ2EV0 type=del size=1046485 2023-12-03T11:48:41.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.978+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.0 size=1048576 2023-12-03T11:48:41.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.980+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.1 size=1048576 2023-12-03T11:48:41.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.981+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.2 size=704247 2023-12-03T11:48:41.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.981+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=IMXRJAQ80IL5 tag=QG15ZR4K55LL type=write size=3849975 2023-12-03T11:48:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.982+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~Z621AQS66NUI.0 size=905808 2023-12-03T11:48:41.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.984+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=Z621AQS66NUI tag=HTYVMLDT7XWX type=write size=1954384 2023-12-03T11:48:41.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.985+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=K44SUXVWCZX3 type=write size=571126 2023-12-03T11:48:41.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.986+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~KHODDFPHYXA8.0 size=1048576 2023-12-03T11:48:41.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.987+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~KHODDFPHYXA8.1 size=528855 2023-12-03T11:48:41.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.988+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=KHODDFPHYXA8 tag=TFRRBONQGKW5 type=write size=2626007 2023-12-03T11:48:41.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.990+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=N5CI3Q2PBKIV type=del size=421750 2023-12-03T11:48:41.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.991+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=L8DLA5U7JE4F type=del 2023-12-03T11:48:41.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.992+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=U02SAS0K59W4 type=del size=840855 2023-12-03T11:48:41.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.993+0000 7f24574f5780 -1 < failed to complete operation key=S73M6UFMX5NH tag=HYLEY960MC50 type=del size=2843137: (22) Invalid argument 2023-12-03T11:48:41.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.994+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=F00PU3C2YTKG type=del size=3080733 2023-12-03T11:48:41.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.995+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~L145W8SXOKIX.0 size=483484 2023-12-03T11:48:41.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.997+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=L145W8SXOKIX tag=065Q8692NHNV type=write size=1532060 2023-12-03T11:48:42.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:41.999+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=HVECUABXCCX1 type=del size=347996 2023-12-03T11:48:42.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.000+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~NEYREFJOLJG9.0 size=1048576 2023-12-03T11:48:42.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.001+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~NEYREFJOLJG9.1 size=773382 2023-12-03T11:48:42.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.003+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=NEYREFJOLJG9 tag=EXBHB68XKFRL type=write size=2870534 2023-12-03T11:48:42.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.005+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~DQ7I6TC02WPD.0 size=911265 2023-12-03T11:48:42.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.005+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=DQ7I6TC02WPD tag=OUBUR3QXNCQ7 type=write size=1959841 2023-12-03T11:48:42.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.006+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=A84I9QON73BN type=del 2023-12-03T11:48:42.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.007+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=ZQJXV1B1XR2Q type=del size=477208 2023-12-03T11:48:42.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.009+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=37EH1YM993KE type=del 2023-12-03T11:48:42.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.011+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~PECJ7I14I44S.0 size=927459 2023-12-03T11:48:42.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.011+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=PECJ7I14I44S tag=ULE0MFVXSIZI type=write size=1976035 2023-12-03T11:48:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.012+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~50VQ0DT96EO1.0 size=1048576 2023-12-03T11:48:42.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.013+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~50VQ0DT96EO1.1 size=627383 2023-12-03T11:48:42.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.015+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=50VQ0DT96EO1 tag=S39R75FW4S38 type=write size=2724535 2023-12-03T11:48:42.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.016+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=ZVME20TOQ7P9 type=write size=514012 2023-12-03T11:48:42.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.017+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=C8GPEB0V0HMT type=del 2023-12-03T11:48:42.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.018+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=KGU6IS8A5ED3 type=del size=1152025 2023-12-03T11:48:42.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.021+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=J03UNNOGZRW0 type=del 2023-12-03T11:48:42.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.023+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.0 size=1048576 2023-12-03T11:48:42.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.024+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.1 size=1048576 2023-12-03T11:48:42.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.025+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.2 size=712279 2023-12-03T11:48:42.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.025+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=R7UK0U3H5I51 tag=WWYDEAZ8S8CI type=write size=3858007 2023-12-03T11:48:42.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.027+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.0 size=1048576 2023-12-03T11:48:42.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.028+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.1 size=1048576 2023-12-03T11:48:42.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.029+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~LVOHPUSF1SYZ.2 size=205445 2023-12-03T11:48:42.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.031+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=LVOHPUSF1SYZ tag=4T2SB32X9UAZ type=write size=3351173 2023-12-03T11:48:42.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.033+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~J7IMIU1VOIMW.0 size=7286 2023-12-03T11:48:42.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.033+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=J7IMIU1VOIMW tag=9YPRHOG7IT46 type=write size=1055862 2023-12-03T11:48:42.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.034+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=L4W51ZT2UIXH type=del 2023-12-03T11:48:42.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.035+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=Q0QOS1SDBHRR type=del size=2182818 2023-12-03T11:48:42.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.036+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=PA1YANYO37EZ type=del 2023-12-03T11:48:42.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.038+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=76GOCLSER9D4 type=del size=3726571 2023-12-03T11:48:42.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.039+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=O7ZE3I7AWPD7 type=del 2023-12-03T11:48:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.040+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=CZW6IDFRCG70 type=del size=4105612 2023-12-03T11:48:42.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.041+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.0 size=1048576 2023-12-03T11:48:42.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.042+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.1 size=1048576 2023-12-03T11:48:42.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.043+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~IMXRJAQ80IL5.2 size=704247 2023-12-03T11:48:42.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.045+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=IMXRJAQ80IL5 tag=QG15ZR4K55LL type=write size=3849975 2023-12-03T11:48:42.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.046+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=PN662YHIT521 type=write size=304735 2023-12-03T11:48:42.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.048+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=K44SUXVWCZX3 type=write size=571126 2023-12-03T11:48:42.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.049+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=FTZB8ZC4FT3R type=del size=2270678 2023-12-03T11:48:42.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.050+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=N5CI3Q2PBKIV type=del size=421750 2023-12-03T11:48:42.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.052+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.0 size=1048576 2023-12-03T11:48:42.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.053+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.1 size=1048576 2023-12-03T11:48:42.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.054+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.2 size=958826 2023-12-03T11:48:42.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.054+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=40HRK00EHD73 tag=Z2776N70S2VD type=write size=4104554 2023-12-03T11:48:42.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.057+0000 7f24574f5780 1 < suggested remove operation key=LS4OEAGZWQXC tag=U02SAS0K59W4 type=del 2023-12-03T11:48:42.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.059+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.0 size=1048576 2023-12-03T11:48:42.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.060+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.1 size=1048576 2023-12-03T11:48:42.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.061+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.2 size=906154 2023-12-03T11:48:42.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.061+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=FRGHD8LE5AGB tag=IQ0C1QHKIV58 type=write size=4051882 2023-12-03T11:48:42.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.062+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=F00PU3C2YTKG type=del 2023-12-03T11:48:42.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.065+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~FRYOURN3KH6N.0 size=655111 2023-12-03T11:48:42.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.065+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=FRYOURN3KH6N tag=ADJOWACB2ZHX type=write size=1703687 2023-12-03T11:48:42.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.066+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=HVECUABXCCX1 type=del 2023-12-03T11:48:42.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.068+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.0 size=1048576 2023-12-03T11:48:42.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.069+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.1 size=1048576 2023-12-03T11:48:42.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.070+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.2 size=578895 2023-12-03T11:48:42.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.070+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=Y2QEL7BBEZV6 tag=SPDT9Y1DVAH6 type=write size=3724623 2023-12-03T11:48:42.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.071+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~DQ7I6TC02WPD.0 size=911265 2023-12-03T11:48:42.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.073+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=DQ7I6TC02WPD tag=OUBUR3QXNCQ7 type=write size=1959841 2023-12-03T11:48:42.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.074+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=91Z3RH4SU2L5 type=del size=1978096 2023-12-03T11:48:42.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.075+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=ZQJXV1B1XR2Q type=del 2023-12-03T11:48:42.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.077+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.0 size=1048576 2023-12-03T11:48:42.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.078+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.1 size=1048576 2023-12-03T11:48:42.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.079+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.2 size=980491 2023-12-03T11:48:42.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.079+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=ML9FZF4LII0M tag=LWOTPRVR8L0C type=write size=4126219 2023-12-03T11:48:42.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.080+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~PECJ7I14I44S.0 size=927459 2023-12-03T11:48:42.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.082+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=PECJ7I14I44S tag=ULE0MFVXSIZI type=write size=1976035 2023-12-03T11:48:42.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.084+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~8K5HVXRYDHOG.0 size=1048576 2023-12-03T11:48:42.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.085+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~8K5HVXRYDHOG.1 size=28418 2023-12-03T11:48:42.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.085+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=8K5HVXRYDHOG tag=5YQPNEIYU5XM type=write size=2125570 2023-12-03T11:48:42.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.086+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=ZVME20TOQ7P9 type=write size=514012 2023-12-03T11:48:42.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.088+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=YB65IF2N0EYL type=del size=3202568 2023-12-03T11:48:42.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.089+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=KGU6IS8A5ED3 type=del 2023-12-03T11:48:42.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.090+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=2WP6GIV0CE6N type=del size=1114235 2023-12-03T11:48:42.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.091+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.0 size=1048576 2023-12-03T11:48:42.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.092+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.1 size=1048576 2023-12-03T11:48:42.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.093+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~R7UK0U3H5I51.2 size=712279 2023-12-03T11:48:42.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.097+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=R7UK0U3H5I51 tag=WWYDEAZ8S8CI type=write size=3858007 2023-12-03T11:48:42.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.099+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=GQXYTY7AQJIF type=del size=2897340 2023-12-03T11:48:42.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.100+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~J7IMIU1VOIMW.0 size=7286 2023-12-03T11:48:42.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.102+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=J7IMIU1VOIMW tag=9YPRHOG7IT46 type=write size=1055862 2023-12-03T11:48:42.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.104+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.0 size=1048576 2023-12-03T11:48:42.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.105+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.1 size=1048576 2023-12-03T11:48:42.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.106+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.2 size=349443 2023-12-03T11:48:42.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.106+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=1BOB74FYORRA tag=UYDKII3O1FAH type=write size=3495171 2023-12-03T11:48:42.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.108+0000 7f24574f5780 1 < suggested update operation key=35U7YCDYMX46 tag=Q0QOS1SDBHRR type=del size=2182818 2023-12-03T11:48:42.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.110+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.0 size=1048576 2023-12-03T11:48:42.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.111+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.1 size=1048576 2023-12-03T11:48:42.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.112+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.2 size=201100 2023-12-03T11:48:42.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.112+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=7Z028BNE023K tag=P1RUMFXSUAYL type=write size=3346828 2023-12-03T11:48:42.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.113+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=76GOCLSER9D4 type=del 2023-12-03T11:48:42.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.114+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=Z95NYETKCSJB type=del size=2573477 2023-12-03T11:48:42.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.115+0000 7f24574f5780 -1 < failed to complete operation key=LS4OEAGZWQXC tag=CZW6IDFRCG70 type=del size=4105612: (22) Invalid argument 2023-12-03T11:48:42.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.116+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=K5PGE2EJMV7H type=del size=2354572 2023-12-03T11:48:42.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.117+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=PN662YHIT521 type=write size=304735 2023-12-03T11:48:42.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.118+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=GAZU1DA21UPN type=write size=807467 2023-12-03T11:48:42.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.121+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=FTZB8ZC4FT3R type=del 2023-12-03T11:48:42.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.122+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=XOHEJN9UV21T type=del size=589667 2023-12-03T11:48:42.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.124+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.0 size=1048576 2023-12-03T11:48:42.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.124+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.1 size=1048576 2023-12-03T11:48:42.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.125+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~40HRK00EHD73.2 size=958826 2023-12-03T11:48:42.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.127+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=40HRK00EHD73 tag=Z2776N70S2VD type=write size=4104554 2023-12-03T11:48:42.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.128+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=SSGRZO874EX7 type=write size=981806 2023-12-03T11:48:42.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.129+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.0 size=1048576 2023-12-03T11:48:42.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.131+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.1 size=1048576 2023-12-03T11:48:42.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.132+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~FRGHD8LE5AGB.2 size=906154 2023-12-03T11:48:42.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.134+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=FRGHD8LE5AGB tag=IQ0C1QHKIV58 type=write size=4051882 2023-12-03T11:48:42.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.135+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=NR4USNEVFUDT type=write size=1037134 2023-12-03T11:48:42.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.136+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~FRYOURN3KH6N.0 size=655111 2023-12-03T11:48:42.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.138+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=FRYOURN3KH6N tag=ADJOWACB2ZHX type=write size=1703687 2023-12-03T11:48:42.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.140+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~S7NI2JEXT0WU.0 size=1048576 2023-12-03T11:48:42.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.141+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~S7NI2JEXT0WU.1 size=670316 2023-12-03T11:48:42.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.141+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=S7NI2JEXT0WU tag=T7RFIE40O63F type=write size=2767468 2023-12-03T11:48:42.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.142+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.0 size=1048576 2023-12-03T11:48:42.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.143+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.1 size=1048576 2023-12-03T11:48:42.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.144+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~Y2QEL7BBEZV6.2 size=578895 2023-12-03T11:48:42.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.146+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=Y2QEL7BBEZV6 tag=SPDT9Y1DVAH6 type=write size=3724623 2023-12-03T11:48:42.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.147+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=75Y1KQEX272R type=write size=393400 2023-12-03T11:48:42.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.148+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=91Z3RH4SU2L5 type=del 2023-12-03T11:48:42.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.150+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~YEWKQHOJXIBB.0 size=830203 2023-12-03T11:48:42.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.150+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=YEWKQHOJXIBB tag=ZJ0F8GSWBQD0 type=write size=1878779 2023-12-03T11:48:42.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.152+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.0 size=1048576 2023-12-03T11:48:42.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.153+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.1 size=1048576 2023-12-03T11:48:42.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.154+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ML9FZF4LII0M.2 size=980491 2023-12-03T11:48:42.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.157+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=ML9FZF4LII0M tag=LWOTPRVR8L0C type=write size=4126219 2023-12-03T11:48:42.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.158+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=OXXIPN2ECUTM type=write size=944245 2023-12-03T11:48:42.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.160+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~8K5HVXRYDHOG.0 size=1048576 2023-12-03T11:48:42.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.161+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~8K5HVXRYDHOG.1 size=28418 2023-12-03T11:48:42.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.162+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=8K5HVXRYDHOG tag=5YQPNEIYU5XM type=write size=2125570 2023-12-03T11:48:42.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.164+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=2PUAYXTUIU0R type=del size=2056187 2023-12-03T11:48:42.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.166+0000 7f24574f5780 1 < suggested remove operation key=RABUR02D2ZMD tag=YB65IF2N0EYL type=del 2023-12-03T11:48:42.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.167+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=V3MXL0CPO1Q6 type=write size=685490 2023-12-03T11:48:42.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.168+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=2WP6GIV0CE6N type=del 2023-12-03T11:48:42.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.169+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=S4246RYPAW0U type=del size=76893 2023-12-03T11:48:42.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.171+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=GQXYTY7AQJIF type=del 2023-12-03T11:48:42.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.173+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.0 size=1048576 2023-12-03T11:48:42.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.174+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.1 size=1048576 2023-12-03T11:48:42.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.176+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.2 size=354442 2023-12-03T11:48:42.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.176+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=870I2I5A07HI tag=H8JJ4SZW7QLV type=write size=3500170 2023-12-03T11:48:42.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.177+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.0 size=1048576 2023-12-03T11:48:42.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.179+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.1 size=1048576 2023-12-03T11:48:42.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.180+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~1BOB74FYORRA.2 size=349443 2023-12-03T11:48:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.182+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=1BOB74FYORRA tag=UYDKII3O1FAH type=write size=3495171 2023-12-03T11:48:42.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.185+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~M3QC3EHLE77F.0 size=1048576 2023-12-03T11:48:42.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.186+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~M3QC3EHLE77F.1 size=244414 2023-12-03T11:48:42.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.186+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=M3QC3EHLE77F tag=2UCX1MKB0UQ2 type=write size=2341566 2023-12-03T11:48:42.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.187+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.0 size=1048576 2023-12-03T11:48:42.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.187+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.1 size=1048576 2023-12-03T11:48:42.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.187+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~7Z028BNE023K.2 size=201100 2023-12-03T11:48:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.189+0000 7f24574f5780 1 < canceled multipart upload key=RABUR02D2ZMD upload=7Z028BNE023K tag=P1RUMFXSUAYL type=write size=3346828 2023-12-03T11:48:42.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.190+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=GBCA43EZ91J3 type=write size=381629 2023-12-03T11:48:42.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.193+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=Z95NYETKCSJB type=del 2023-12-03T11:48:42.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.194+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=FXUGH5VWE4ZS type=del size=3287870 2023-12-03T11:48:42.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.195+0000 7f24574f5780 1 < canceled operation key=MY3K9M4QRS3L tag=K5PGE2EJMV7H type=del size=2354572 2023-12-03T11:48:42.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.196+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=O5PZJKJ07E63 type=del size=1620025 2023-12-03T11:48:42.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.198+0000 7f24574f5780 1 < completed write operation key=XRHVYVADRWET tag=GAZU1DA21UPN type=write size=807467 2023-12-03T11:48:42.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.200+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.0 size=1048576 2023-12-03T11:48:42.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.200+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.1 size=1048576 2023-12-03T11:48:42.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.201+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.2 size=633216 2023-12-03T11:48:42.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.201+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=UPYCYS0BVJ7J tag=ZARJLDVDAXR3 type=write size=3778944 2023-12-03T11:48:42.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.203+0000 7f24574f5780 1 < canceled operation key=QVLPCWC4T8N8 tag=XOHEJN9UV21T type=del size=589667 2023-12-03T11:48:42.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.205+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~F0KB14CQTNTR.0 size=1048576 2023-12-03T11:48:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.205+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~F0KB14CQTNTR.1 size=581026 2023-12-03T11:48:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.205+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=F0KB14CQTNTR tag=6OCY2DHVILLZ type=write size=2678178 2023-12-03T11:48:42.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.207+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=SSGRZO874EX7 type=write size=981806 2023-12-03T11:48:42.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.208+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=V3GZ0TR7FHMX type=del size=2631319 2023-12-03T11:48:42.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.209+0000 7f24574f5780 -1 < failed to complete operation key=3EOEF9UMATIE tag=NR4USNEVFUDT type=write size=1037134: (22) Invalid argument 2023-12-03T11:48:42.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.210+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=HTGUO5QOQUNX type=del size=1923570 2023-12-03T11:48:42.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.211+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~S7NI2JEXT0WU.0 size=1048576 2023-12-03T11:48:42.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.212+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~S7NI2JEXT0WU.1 size=670316 2023-12-03T11:48:42.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.214+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=S7NI2JEXT0WU tag=T7RFIE40O63F type=write size=2767468 2023-12-03T11:48:42.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.215+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=46MXFBBMBRWD type=write size=278257 2023-12-03T11:48:42.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.216+0000 7f24574f5780 1 < completed write operation key=XRHVYVADRWET tag=75Y1KQEX272R type=write size=393400 2023-12-03T11:48:42.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.217+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=DBIOV0MPKH0A type=del size=3885605 2023-12-03T11:48:42.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.219+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~YEWKQHOJXIBB.0 size=830203 2023-12-03T11:48:42.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.221+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=YEWKQHOJXIBB tag=ZJ0F8GSWBQD0 type=write size=1878779 2023-12-03T11:48:42.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.222+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=UAH0IJZZCJIV type=del size=3313177 2023-12-03T11:48:42.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.223+0000 7f24574f5780 1 < completed write operation key=S7Q5TCM499H1 tag=OXXIPN2ECUTM type=write size=944245 2023-12-03T11:48:42.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.225+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~K0FIWNUEP3T7.0 size=1048576 2023-12-03T11:48:42.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.226+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~K0FIWNUEP3T7.1 size=447085 2023-12-03T11:48:42.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.226+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=K0FIWNUEP3T7 tag=859EOPZRD0Z4 type=write size=2544237 2023-12-03T11:48:42.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.228+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=2PUAYXTUIU0R type=del size=2056187 2023-12-03T11:48:42.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.229+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=UXCLE4LFLQKH type=del size=114200 2023-12-03T11:48:42.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.231+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=V3MXL0CPO1Q6 type=write size=685490 2023-12-03T11:48:42.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.233+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.0 size=1048576 2023-12-03T11:48:42.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.234+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.1 size=1048576 2023-12-03T11:48:42.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.234+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.2 size=835718 2023-12-03T11:48:42.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.234+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=NTHXFQU44X38 tag=4QE011VPJNLL type=write size=3981446 2023-12-03T11:48:42.237 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.236+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=S4246RYPAW0U type=del 2023-12-03T11:48:42.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.237+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=NIDBYFDQTTTU type=del size=590530 2023-12-03T11:48:42.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.238+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.0 size=1048576 2023-12-03T11:48:42.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.239+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.1 size=1048576 2023-12-03T11:48:42.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.240+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~870I2I5A07HI.2 size=354442 2023-12-03T11:48:42.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.242+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=870I2I5A07HI tag=H8JJ4SZW7QLV type=write size=3500170 2023-12-03T11:48:42.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.243+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=S4CCZYOV0MM9 type=del size=2872455 2023-12-03T11:48:42.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.244+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~M3QC3EHLE77F.0 size=1048576 2023-12-03T11:48:42.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.245+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~M3QC3EHLE77F.1 size=244414 2023-12-03T11:48:42.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.247+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=M3QC3EHLE77F tag=2UCX1MKB0UQ2 type=write size=2341566 2023-12-03T11:48:42.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.249+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=16PRS3WCZZTD type=del size=2792584 2023-12-03T11:48:42.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.250+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=GBCA43EZ91J3 type=write size=381629 2023-12-03T11:48:42.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.252+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.0 size=1048576 2023-12-03T11:48:42.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.253+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.1 size=1048576 2023-12-03T11:48:42.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.254+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.2 size=760724 2023-12-03T11:48:42.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.254+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=SU2AKDV885LT tag=ESFC30Q8KXYZ type=write size=3906452 2023-12-03T11:48:42.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.255+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=FXUGH5VWE4ZS type=del 2023-12-03T11:48:42.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.256+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=5P7L1G3582HT type=del size=2036376 2023-12-03T11:48:42.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.257+0000 7f24574f5780 1 < canceled operation key=863EOZDYRWH3 tag=O5PZJKJ07E63 type=del size=1620025 2023-12-03T11:48:42.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.260+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~ECM41FNW7IJL.0 size=1048576 2023-12-03T11:48:42.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.261+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~ECM41FNW7IJL.1 size=320960 2023-12-03T11:48:42.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.261+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=ECM41FNW7IJL tag=I925Y7WW0PMD type=write size=2418112 2023-12-03T11:48:42.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.262+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.0 size=1048576 2023-12-03T11:48:42.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.263+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.1 size=1048576 2023-12-03T11:48:42.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.263+0000 7f24574f5780 1 < canceled part key=_multipart_XO42YZR60KQE.2~UPYCYS0BVJ7J.2 size=633216 2023-12-03T11:48:42.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.265+0000 7f24574f5780 1 < canceled multipart upload key=XO42YZR60KQE upload=UPYCYS0BVJ7J tag=ZARJLDVDAXR3 type=write size=3778944 2023-12-03T11:48:42.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.267+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.0 size=1048576 2023-12-03T11:48:42.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.268+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.1 size=1048576 2023-12-03T11:48:42.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.269+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.2 size=565057 2023-12-03T11:48:42.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.269+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=9SVR3POZ1RVM tag=7P1Q4O2XW491 type=write size=3710785 2023-12-03T11:48:42.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.270+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~F0KB14CQTNTR.0 size=1048576 2023-12-03T11:48:42.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.271+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~F0KB14CQTNTR.1 size=581026 2023-12-03T11:48:42.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.273+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=F0KB14CQTNTR tag=6OCY2DHVILLZ type=write size=2678178 2023-12-03T11:48:42.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.274+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=VNW8DD2H026A type=write size=994105 2023-12-03T11:48:42.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.275+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=V3GZ0TR7FHMX type=del 2023-12-03T11:48:42.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.276+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=PRH2GY7J1WS7 type=del size=3490493 2023-12-03T11:48:42.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.277+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=HTGUO5QOQUNX type=del 2023-12-03T11:48:42.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.280+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~BV92GRIFOZAC.0 size=1048576 2023-12-03T11:48:42.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.281+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~BV92GRIFOZAC.1 size=408110 2023-12-03T11:48:42.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.281+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=BV92GRIFOZAC tag=Y7LTW6L0MF4G type=write size=2505262 2023-12-03T11:48:42.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.283+0000 7f24574f5780 1 < suggested remove operation key=MH1SRAATMNXB tag=46MXFBBMBRWD type=write 2023-12-03T11:48:42.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.284+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=C4PBRXIKDT7Z type=del size=3121129 2023-12-03T11:48:42.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.287+0000 7f24574f5780 1 < suggested remove operation key=SUBILSL92D96 tag=DBIOV0MPKH0A type=del 2023-12-03T11:48:42.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.288+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=2YUKOOLVIQNI type=write size=670974 2023-12-03T11:48:42.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.290+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=UAH0IJZZCJIV type=del 2023-12-03T11:48:42.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.291+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=DU82K4A85YDV type=write size=326896 2023-12-03T11:48:42.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.292+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~K0FIWNUEP3T7.0 size=1048576 2023-12-03T11:48:42.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.292+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~K0FIWNUEP3T7.1 size=447085 2023-12-03T11:48:42.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.294+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=K0FIWNUEP3T7 tag=859EOPZRD0Z4 type=write size=2544237 2023-12-03T11:48:42.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.295+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=B12M01KPS1FD type=del size=3818292 2023-12-03T11:48:42.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.296+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=UXCLE4LFLQKH type=del 2023-12-03T11:48:42.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.297+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=TW20OR9107IX type=write size=303801 2023-12-03T11:48:42.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.298+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.0 size=1048576 2023-12-03T11:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.299+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.1 size=1048576 2023-12-03T11:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.299+0000 7f24574f5780 1 < canceled part key=_multipart_XRHVYVADRWET.2~NTHXFQU44X38.2 size=835718 2023-12-03T11:48:42.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.301+0000 7f24574f5780 1 < canceled multipart upload key=XRHVYVADRWET upload=NTHXFQU44X38 tag=4QE011VPJNLL type=write size=3981446 2023-12-03T11:48:42.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.303+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.0 size=1048576 2023-12-03T11:48:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.304+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.1 size=1048576 2023-12-03T11:48:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.305+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.2 size=282953 2023-12-03T11:48:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.305+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=H4BB2LNPYC03 tag=R06RO21LPH9K type=write size=3428681 2023-12-03T11:48:42.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.307+0000 7f24574f5780 1 < suggested remove operation key=LS4OEAGZWQXC tag=NIDBYFDQTTTU type=del 2023-12-03T11:48:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.308+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=BMKHWFADK6EH type=del size=927676 2023-12-03T11:48:42.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.310+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=S4CCZYOV0MM9 type=del size=2872455 2023-12-03T11:48:42.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.312+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.0 size=1048576 2023-12-03T11:48:42.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.313+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.1 size=1048576 2023-12-03T11:48:42.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.313+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.2 size=282887 2023-12-03T11:48:42.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.313+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=0WIUTOW0I4NL tag=2CKWAPVABR6I type=write size=3428615 2023-12-03T11:48:42.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.315+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=16PRS3WCZZTD type=del 2023-12-03T11:48:42.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.316+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=TBV6M5ODUS4S type=write size=430565 2023-12-03T11:48:42.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.317+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.0 size=1048576 2023-12-03T11:48:42.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.318+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.1 size=1048576 2023-12-03T11:48:42.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.319+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~SU2AKDV885LT.2 size=760724 2023-12-03T11:48:42.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.321+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=SU2AKDV885LT tag=ESFC30Q8KXYZ type=write size=3906452 2023-12-03T11:48:42.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.323+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~R7PAKEVZ47J2.0 size=1048576 2023-12-03T11:48:42.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.324+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~R7PAKEVZ47J2.1 size=944370 2023-12-03T11:48:42.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.324+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=R7PAKEVZ47J2 tag=TUT54XD0DKKI type=write size=3041522 2023-12-03T11:48:42.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.325+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=5P7L1G3582HT type=del 2023-12-03T11:48:42.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.326+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=EPRZH1OT82I7 type=del size=442154 2023-12-03T11:48:42.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.327+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~ECM41FNW7IJL.0 size=1048576 2023-12-03T11:48:42.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.328+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~ECM41FNW7IJL.1 size=320960 2023-12-03T11:48:42.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.330+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=ECM41FNW7IJL tag=I925Y7WW0PMD type=write size=2418112 2023-12-03T11:48:42.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.332+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=G3JGNUSPG5CP type=del size=1380522 2023-12-03T11:48:42.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.332+0000 7f24574f5780 1 < canceled part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.0 size=1048576 2023-12-03T11:48:42.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.333+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.1 size=1048576 2023-12-03T11:48:42.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.334+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~9SVR3POZ1RVM.2 size=565057 2023-12-03T11:48:42.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.336+0000 7f24574f5780 1 < canceled multipart upload key=MH1SRAATMNXB upload=9SVR3POZ1RVM tag=7P1Q4O2XW491 type=write size=3710785 2023-12-03T11:48:42.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.337+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=GXJMA5OX9KIO type=del size=639382 2023-12-03T11:48:42.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.338+0000 7f24574f5780 1 < canceled operation key=35U7YCDYMX46 tag=VNW8DD2H026A type=write size=994105 2023-12-03T11:48:42.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.340+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.0 size=1048576 2023-12-03T11:48:42.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.341+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.1 size=1048576 2023-12-03T11:48:42.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.342+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.2 size=404048 2023-12-03T11:48:42.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.342+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=JZB7FG0NHNL4 tag=967YI50H55CI type=write size=3549776 2023-12-03T11:48:42.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.343+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=PRH2GY7J1WS7 type=del 2023-12-03T11:48:42.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.345+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.0 size=1048576 2023-12-03T11:48:42.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.346+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.1 size=1048576 2023-12-03T11:48:42.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.347+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.2 size=516168 2023-12-03T11:48:42.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.347+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=6NU9PRBPNSWZ tag=TDT2EN2WE1BJ type=write size=3661896 2023-12-03T11:48:42.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.349+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~BV92GRIFOZAC.0 size=1048576 2023-12-03T11:48:42.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.350+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~BV92GRIFOZAC.1 size=408110 2023-12-03T11:48:42.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.353+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=BV92GRIFOZAC tag=Y7LTW6L0MF4G type=write size=2505262 2023-12-03T11:48:42.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.354+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=KARJ79UOAQK7 type=write size=300309 2023-12-03T11:48:42.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.355+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=C4PBRXIKDT7Z type=del size=3121129 2023-12-03T11:48:42.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.356+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=YHA0HJ7FEOBT type=write size=390967 2023-12-03T11:48:42.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.358+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=2YUKOOLVIQNI type=write size=670974 2023-12-03T11:48:42.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.360+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~ANNKTQWVMTQK.0 size=61660 2023-12-03T11:48:42.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.360+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=ANNKTQWVMTQK tag=HZ2M6K0W7IFO type=write size=1110236 2023-12-03T11:48:42.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.362+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=DU82K4A85YDV type=write size=326896 2023-12-03T11:48:42.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.363+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=1L9XC92H9F2Y type=del size=1200418 2023-12-03T11:48:42.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.364+0000 7f24574f5780 -1 < failed to complete operation key=BV8C0VSJ1GTW tag=B12M01KPS1FD type=del size=3818292: (22) Invalid argument 2023-12-03T11:48:42.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.366+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.0 size=1048576 2023-12-03T11:48:42.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.367+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.1 size=1048576 2023-12-03T11:48:42.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.368+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.2 size=737034 2023-12-03T11:48:42.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.368+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=EDQSUKPXDRTL tag=X49H1PUWHD8B type=write size=3882762 2023-12-03T11:48:42.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.371+0000 7f24574f5780 1 < suggested remove operation key=PGTGVLK4DBN0 tag=TW20OR9107IX type=write 2023-12-03T11:48:42.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.372+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=O2OBD6VZOQ9R type=write size=207002 2023-12-03T11:48:42.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.373+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.0 size=1048576 2023-12-03T11:48:42.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.374+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.1 size=1048576 2023-12-03T11:48:42.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.375+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~H4BB2LNPYC03.2 size=282953 2023-12-03T11:48:42.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.377+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=H4BB2LNPYC03 tag=R06RO21LPH9K type=write size=3428681 2023-12-03T11:48:42.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.378+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=ED0JNQ6GI2VM type=del size=2624101 2023-12-03T11:48:42.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.380+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=BMKHWFADK6EH type=del 2023-12-03T11:48:42.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.382+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~M8L5L0X5YX9J.0 size=1048576 2023-12-03T11:48:42.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.383+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~M8L5L0X5YX9J.1 size=1042012 2023-12-03T11:48:42.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.383+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=M8L5L0X5YX9J tag=KFJAKJIMS652 type=write size=3139164 2023-12-03T11:48:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.384+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.0 size=1048576 2023-12-03T11:48:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.385+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.1 size=1048576 2023-12-03T11:48:42.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.386+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~0WIUTOW0I4NL.2 size=282887 2023-12-03T11:48:42.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.388+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=0WIUTOW0I4NL tag=2CKWAPVABR6I type=write size=3428615 2023-12-03T11:48:42.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.389+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=ZGIHTB0KCJVQ type=del size=776114 2023-12-03T11:48:42.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.390+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=TBV6M5ODUS4S type=write size=430565 2023-12-03T11:48:42.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.392+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~CJFC935FX8X5.0 size=524969 2023-12-03T11:48:42.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.392+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=CJFC935FX8X5 tag=837AJ0FKVPWL type=write size=1573545 2023-12-03T11:48:42.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.393+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~R7PAKEVZ47J2.0 size=1048576 2023-12-03T11:48:42.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.393+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~R7PAKEVZ47J2.1 size=944370 2023-12-03T11:48:42.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.396+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=R7PAKEVZ47J2 tag=TUT54XD0DKKI type=write size=3041522 2023-12-03T11:48:42.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.397+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=ICWJYYRS6N25 type=del size=4024675 2023-12-03T11:48:42.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.398+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=EPRZH1OT82I7 type=del 2023-12-03T11:48:42.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.400+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=LB67P32ZRQPV type=del size=3433779 2023-12-03T11:48:42.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.401+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=G3JGNUSPG5CP type=del 2023-12-03T11:48:42.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.402+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=NWF1E4P16CWD type=write size=769979 2023-12-03T11:48:42.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.403+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=GXJMA5OX9KIO type=del 2023-12-03T11:48:42.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.404+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=5AK1DTP5W4ZG type=del size=1832881 2023-12-03T11:48:42.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.406+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.0 size=1048576 2023-12-03T11:48:42.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.406+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.1 size=1048576 2023-12-03T11:48:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.407+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~JZB7FG0NHNL4.2 size=404048 2023-12-03T11:48:42.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.409+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=JZB7FG0NHNL4 tag=967YI50H55CI type=write size=3549776 2023-12-03T11:48:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.411+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=UMGCRG08VBSZ type=del size=2698682 2023-12-03T11:48:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.412+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.0 size=1048576 2023-12-03T11:48:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.412+0000 7f24574f5780 1 < canceled part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.1 size=1048576 2023-12-03T11:48:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.413+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~6NU9PRBPNSWZ.2 size=516168 2023-12-03T11:48:42.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.414+0000 7f24574f5780 1 < canceled multipart upload key=LS4OEAGZWQXC upload=6NU9PRBPNSWZ tag=TDT2EN2WE1BJ type=write size=3661896 2023-12-03T11:48:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.416+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=1ICRVZ3Y62XW type=del size=3261220 2023-12-03T11:48:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.417+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=KARJ79UOAQK7 type=write size=300309 2023-12-03T11:48:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.418+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=IXBCKL9Q61AW type=write size=390157 2023-12-03T11:48:42.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.419+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=YHA0HJ7FEOBT type=write size=390967 2023-12-03T11:48:42.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.420+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=SIXHMG9BHJXS type=write size=933851 2023-12-03T11:48:42.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.422+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~ANNKTQWVMTQK.0 size=61660 2023-12-03T11:48:42.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.424+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=ANNKTQWVMTQK tag=HZ2M6K0W7IFO type=write size=1110236 2023-12-03T11:48:42.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.425+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=J7F5OVHIW3J0 type=del size=3434022 2023-12-03T11:48:42.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.426+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=1L9XC92H9F2Y type=del 2023-12-03T11:48:42.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.427+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=81FKRWDLSCA3 type=del size=1213013 2023-12-03T11:48:42.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.428+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.0 size=1048576 2023-12-03T11:48:42.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.429+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.1 size=1048576 2023-12-03T11:48:42.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.430+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~EDQSUKPXDRTL.2 size=737034 2023-12-03T11:48:42.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.432+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=EDQSUKPXDRTL tag=X49H1PUWHD8B type=write size=3882762 2023-12-03T11:48:42.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.433+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=LFZ2ECJ02JSN type=del size=1970274 2023-12-03T11:48:42.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.435+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=O2OBD6VZOQ9R type=write size=207002 2023-12-03T11:48:42.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.437+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.0 size=1048576 2023-12-03T11:48:42.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.438+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.1 size=1048576 2023-12-03T11:48:42.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.438+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.2 size=793536 2023-12-03T11:48:42.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.438+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=4H1WW60ZHQJI tag=57C49M3V1NEC type=write size=3939264 2023-12-03T11:48:42.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.441+0000 7f24574f5780 1 < suggested update operation key=1QR0TE9DXU5Z tag=ED0JNQ6GI2VM type=del size=2624101 2023-12-03T11:48:42.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.443+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~76PWJC881414.0 size=1048576 2023-12-03T11:48:42.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.444+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~76PWJC881414.1 size=772450 2023-12-03T11:48:42.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.444+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=76PWJC881414 tag=J65RDZ83J7OE type=write size=2869602 2023-12-03T11:48:42.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.445+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~M8L5L0X5YX9J.0 size=1048576 2023-12-03T11:48:42.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.446+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~M8L5L0X5YX9J.1 size=1042012 2023-12-03T11:48:42.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.448+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=M8L5L0X5YX9J tag=KFJAKJIMS652 type=write size=3139164 2023-12-03T11:48:42.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.450+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~LQE1ME1KSPYV.0 size=1048576 2023-12-03T11:48:42.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.451+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~LQE1ME1KSPYV.1 size=619959 2023-12-03T11:48:42.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.451+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=LQE1ME1KSPYV tag=3DBKLOUG7125 type=write size=2717111 2023-12-03T11:48:42.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.452+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=ZGIHTB0KCJVQ type=del 2023-12-03T11:48:42.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.454+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~VYJB1K39957H.0 size=1048576 2023-12-03T11:48:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.455+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~VYJB1K39957H.1 size=996060 2023-12-03T11:48:42.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.455+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=VYJB1K39957H tag=H33UJL656XM1 type=write size=3093212 2023-12-03T11:48:42.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.456+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~CJFC935FX8X5.0 size=524969 2023-12-03T11:48:42.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.458+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=CJFC935FX8X5 tag=837AJ0FKVPWL type=write size=1573545 2023-12-03T11:48:42.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.459+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=01BBA3U33EKA type=del size=2607550 2023-12-03T11:48:42.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.461+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=ICWJYYRS6N25 type=del 2023-12-03T11:48:42.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.462+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=WO7BQC3R5PPJ type=del size=445624 2023-12-03T11:48:42.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.464+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=LB67P32ZRQPV type=del size=3433779 2023-12-03T11:48:42.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.465+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=KNA71O675Q0Z type=del size=2451431 2023-12-03T11:48:42.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.466+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=NWF1E4P16CWD type=write size=769979 2023-12-03T11:48:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.469+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~FXXBE4S58SH0.0 size=1048576 2023-12-03T11:48:42.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.470+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~FXXBE4S58SH0.1 size=644130 2023-12-03T11:48:42.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.470+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=FXXBE4S58SH0 tag=K490CXAED7IL type=write size=2741282 2023-12-03T11:48:42.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.472+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=5AK1DTP5W4ZG type=del size=1832881 2023-12-03T11:48:42.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.474+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=RTJ06D5TX3V6 type=write size=721815 2023-12-03T11:48:42.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.475+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=UMGCRG08VBSZ type=del 2023-12-03T11:48:42.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.476+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=C9YVOPM2UY0S type=write size=121717 2023-12-03T11:48:42.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.477+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=1ICRVZ3Y62XW type=del 2023-12-03T11:48:42.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.479+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~44RQJNHCV0AQ.0 size=433572 2023-12-03T11:48:42.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.479+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=44RQJNHCV0AQ tag=SE9TQ45U2Y3H type=write size=1482148 2023-12-03T11:48:42.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.481+0000 7f24574f5780 1 < completed write operation key=BV8C0VSJ1GTW tag=IXBCKL9Q61AW type=write size=390157 2023-12-03T11:48:42.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.482+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=ZGF2VRGPB2Q2 type=del size=3083923 2023-12-03T11:48:42.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.483+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=SIXHMG9BHJXS type=write size=933851 2023-12-03T11:48:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.485+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.0 size=1048576 2023-12-03T11:48:42.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.486+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.1 size=1048576 2023-12-03T11:48:42.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.487+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.2 size=654546 2023-12-03T11:48:42.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.487+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=O7AVOIVX0WOM tag=VE7RBA03LOXC type=write size=3800274 2023-12-03T11:48:42.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.488+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=J7F5OVHIW3J0 type=del 2023-12-03T11:48:42.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.490+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=DKJO53BCK02J type=del size=3534104 2023-12-03T11:48:42.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.491+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=81FKRWDLSCA3 type=del 2023-12-03T11:48:42.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.492+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=HQETOOPP4U4E type=del size=3495092 2023-12-03T11:48:42.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.493+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=LFZ2ECJ02JSN type=del 2023-12-03T11:48:42.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.495+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~7PD6XYZPMF95.0 size=1048576 2023-12-03T11:48:42.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.496+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~7PD6XYZPMF95.1 size=906011 2023-12-03T11:48:42.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.496+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=7PD6XYZPMF95 tag=UJP41FVWI9I5 type=write size=3003163 2023-12-03T11:48:42.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.497+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.0 size=1048576 2023-12-03T11:48:42.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.498+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.1 size=1048576 2023-12-03T11:48:42.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.499+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~4H1WW60ZHQJI.2 size=793536 2023-12-03T11:48:42.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.501+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=4H1WW60ZHQJI tag=57C49M3V1NEC type=write size=3939264 2023-12-03T11:48:42.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.502+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=NVJKLKWR20BO type=del size=1677179 2023-12-03T11:48:42.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.504+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~76PWJC881414.0 size=1048576 2023-12-03T11:48:42.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.505+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~76PWJC881414.1 size=772450 2023-12-03T11:48:42.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.508+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=76PWJC881414 tag=J65RDZ83J7OE type=write size=2869602 2023-12-03T11:48:42.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.511+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.0 size=1048576 2023-12-03T11:48:42.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.512+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.1 size=1048576 2023-12-03T11:48:42.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.514+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.2 size=789638 2023-12-03T11:48:42.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.514+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=I76RQUIAHT6V tag=9D7MUETCZTP0 type=write size=3935366 2023-12-03T11:48:42.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.515+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~LQE1ME1KSPYV.0 size=1048576 2023-12-03T11:48:42.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.516+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~LQE1ME1KSPYV.1 size=619959 2023-12-03T11:48:42.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.518+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=LQE1ME1KSPYV tag=3DBKLOUG7125 type=write size=2717111 2023-12-03T11:48:42.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.521+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~IMD00B1HJ9AI.0 size=1048576 2023-12-03T11:48:42.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.522+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~IMD00B1HJ9AI.1 size=82513 2023-12-03T11:48:42.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.522+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=IMD00B1HJ9AI tag=SIEDL3N067KB type=write size=2179665 2023-12-03T11:48:42.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.522+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~VYJB1K39957H.0 size=1048576 2023-12-03T11:48:42.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.523+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~VYJB1K39957H.1 size=996060 2023-12-03T11:48:42.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.525+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=VYJB1K39957H tag=H33UJL656XM1 type=write size=3093212 2023-12-03T11:48:42.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.526+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=3KJTO5UR4WX8 type=write size=469970 2023-12-03T11:48:42.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.527+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=01BBA3U33EKA type=del 2023-12-03T11:48:42.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.528+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=NF6178PSI0KG type=write size=323867 2023-12-03T11:48:42.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.530+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=WO7BQC3R5PPJ type=del 2023-12-03T11:48:42.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.533+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.0 size=1048576 2023-12-03T11:48:42.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.534+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.1 size=1048576 2023-12-03T11:48:42.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.535+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.2 size=69813 2023-12-03T11:48:42.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.535+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=8FV5S64JA5DA tag=2BFJBBAQLLQB type=write size=3215541 2023-12-03T11:48:42.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.536+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=KNA71O675Q0Z type=del 2023-12-03T11:48:42.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.537+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=V14NOLTMV7IM type=del size=1446502 2023-12-03T11:48:42.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.538+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~FXXBE4S58SH0.0 size=1048576 2023-12-03T11:48:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.539+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~FXXBE4S58SH0.1 size=644130 2023-12-03T11:48:42.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.541+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=FXXBE4S58SH0 tag=K490CXAED7IL type=write size=2741282 2023-12-03T11:48:42.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.543+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=D1ERM7N6LP1W type=del size=2483287 2023-12-03T11:48:42.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.544+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=RTJ06D5TX3V6 type=write size=721815 2023-12-03T11:48:42.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.545+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=JZM9VJ6OPWBC type=del size=3781877 2023-12-03T11:48:42.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.546+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=C9YVOPM2UY0S type=write size=121717 2023-12-03T11:48:42.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.548+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=5VKOR8IEKC8F type=del size=3424275 2023-12-03T11:48:42.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.549+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~44RQJNHCV0AQ.0 size=433572 2023-12-03T11:48:42.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.551+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=44RQJNHCV0AQ tag=SE9TQ45U2Y3H type=write size=1482148 2023-12-03T11:48:42.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.552+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=WTNCWW98MGCN type=del size=3937104 2023-12-03T11:48:42.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.553+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=ZGF2VRGPB2Q2 type=del 2023-12-03T11:48:42.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.555+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=O9A8IBB4D4X1 type=write size=746714 2023-12-03T11:48:42.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.556+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.0 size=1048576 2023-12-03T11:48:42.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.557+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.1 size=1048576 2023-12-03T11:48:42.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.558+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~O7AVOIVX0WOM.2 size=654546 2023-12-03T11:48:42.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.560+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=O7AVOIVX0WOM tag=VE7RBA03LOXC type=write size=3800274 2023-12-03T11:48:42.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.562+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~X579L869RBVJ.0 size=937614 2023-12-03T11:48:42.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.562+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=X579L869RBVJ tag=ZL61YU6ZCXRK type=write size=1986190 2023-12-03T11:48:42.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.563+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=DKJO53BCK02J type=del 2023-12-03T11:48:42.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.565+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.0 size=1048576 2023-12-03T11:48:42.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.566+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.1 size=1048576 2023-12-03T11:48:42.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.567+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.2 size=873938 2023-12-03T11:48:42.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.567+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=RKWZ2HZPLZ2H tag=YQNMVSYT6R9S type=write size=4019666 2023-12-03T11:48:42.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.568+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=HQETOOPP4U4E type=del 2023-12-03T11:48:42.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.569+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=BGBX1PSFZADT type=del size=1171424 2023-12-03T11:48:42.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.571+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~7PD6XYZPMF95.0 size=1048576 2023-12-03T11:48:42.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.573+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~7PD6XYZPMF95.1 size=906011 2023-12-03T11:48:42.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.576+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=7PD6XYZPMF95 tag=UJP41FVWI9I5 type=write size=3003163 2023-12-03T11:48:42.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.578+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.0 size=1048576 2023-12-03T11:48:42.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.579+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.1 size=1048576 2023-12-03T11:48:42.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.580+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.2 size=362916 2023-12-03T11:48:42.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.580+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=MUCPWC6PETY1 tag=7UM4ONEV9NHB type=write size=3508644 2023-12-03T11:48:42.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.581+0000 7f24574f5780 1 < canceled operation key=863EOZDYRWH3 tag=NVJKLKWR20BO type=del size=1677179 2023-12-03T11:48:42.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.582+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=3F8M9WNQ8AKZ type=del size=2617940 2023-12-03T11:48:42.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.584+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.0 size=1048576 2023-12-03T11:48:42.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.585+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.1 size=1048576 2023-12-03T11:48:42.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.586+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~I76RQUIAHT6V.2 size=789638 2023-12-03T11:48:42.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.588+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=I76RQUIAHT6V tag=9D7MUETCZTP0 type=write size=3935366 2023-12-03T11:48:42.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.589+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=2M41K5HY4LP1 type=del size=811769 2023-12-03T11:48:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.590+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~IMD00B1HJ9AI.0 size=1048576 2023-12-03T11:48:42.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.591+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~IMD00B1HJ9AI.1 size=82513 2023-12-03T11:48:42.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.593+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=IMD00B1HJ9AI tag=SIEDL3N067KB type=write size=2179665 2023-12-03T11:48:42.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.595+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.0 size=1048576 2023-12-03T11:48:42.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.596+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.1 size=1048576 2023-12-03T11:48:42.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.597+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.2 size=666804 2023-12-03T11:48:42.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.597+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=6KN0YQNOXZL2 tag=63L43VYOKPW6 type=write size=3812532 2023-12-03T11:48:42.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.598+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=3KJTO5UR4WX8 type=write size=469970 2023-12-03T11:48:42.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.600+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~HRAKTYY68MC6.0 size=1048576 2023-12-03T11:48:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.601+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~HRAKTYY68MC6.1 size=96921 2023-12-03T11:48:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.601+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=HRAKTYY68MC6 tag=A421JODFIEC5 type=write size=2194073 2023-12-03T11:48:42.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.602+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=NF6178PSI0KG type=write size=323867 2023-12-03T11:48:42.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.603+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=JYM3LB8CJ0XA type=del size=250809 2023-12-03T11:48:42.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.603+0000 7f24574f5780 1 < canceled part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.0 size=1048576 2023-12-03T11:48:42.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.604+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.1 size=1048576 2023-12-03T11:48:42.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.605+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~8FV5S64JA5DA.2 size=69813 2023-12-03T11:48:42.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.607+0000 7f24574f5780 1 < canceled multipart upload key=XRHVYVADRWET upload=8FV5S64JA5DA tag=2BFJBBAQLLQB type=write size=3215541 2023-12-03T11:48:42.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.609+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~LSQRIR5342AR.0 size=897961 2023-12-03T11:48:42.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.609+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=LSQRIR5342AR tag=VUDYEUOY7QPR type=write size=1946537 2023-12-03T11:48:42.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.610+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=V14NOLTMV7IM type=del 2023-12-03T11:48:42.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.612+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.0 size=1048576 2023-12-03T11:48:42.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.613+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.1 size=1048576 2023-12-03T11:48:42.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.614+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.2 size=755741 2023-12-03T11:48:42.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.614+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=1TFQW21OZXWT tag=0X0CLBYZUHV4 type=write size=3901469 2023-12-03T11:48:42.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.615+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=D1ERM7N6LP1W type=del size=2483287 2023-12-03T11:48:42.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.616+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=5AS05XTO2E7M type=del size=757528 2023-12-03T11:48:42.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.617+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=JZM9VJ6OPWBC type=del 2023-12-03T11:48:42.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.619+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=KHN53C61IEKM type=del size=1561113 2023-12-03T11:48:42.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.620+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=5VKOR8IEKC8F type=del 2023-12-03T11:48:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.622+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=HWS9F1GVINQ3 type=del size=2765752 2023-12-03T11:48:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.623+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=WTNCWW98MGCN type=del 2023-12-03T11:48:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.624+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=0EGVQLL0Z8TK type=del size=2020363 2023-12-03T11:48:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.625+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=O9A8IBB4D4X1 type=write size=746714 2023-12-03T11:48:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.627+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.0 size=1048576 2023-12-03T11:48:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.628+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.1 size=1048576 2023-12-03T11:48:42.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.629+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.2 size=736081 2023-12-03T11:48:42.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.629+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=7BM5QCSJKX1V tag=61BZGUC0W7TK type=write size=3881809 2023-12-03T11:48:42.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.630+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~X579L869RBVJ.0 size=937614 2023-12-03T11:48:42.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.632+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=X579L869RBVJ tag=ZL61YU6ZCXRK type=write size=1986190 2023-12-03T11:48:42.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.633+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=W9MUSTDCNHW3 type=write size=629663 2023-12-03T11:48:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.635+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.0 size=1048576 2023-12-03T11:48:42.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.635+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.1 size=1048576 2023-12-03T11:48:42.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.636+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~RKWZ2HZPLZ2H.2 size=873938 2023-12-03T11:48:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.639+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=RKWZ2HZPLZ2H tag=YQNMVSYT6R9S type=write size=4019666 2023-12-03T11:48:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.641+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~RLFJ0753370T.0 size=33190 2023-12-03T11:48:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.641+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=RLFJ0753370T tag=EWF86U302E8I type=write size=1081766 2023-12-03T11:48:42.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.643+0000 7f24574f5780 1 < suggested remove operation key=6WY91DFIRUN8 tag=BGBX1PSFZADT type=del 2023-12-03T11:48:42.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.645+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=UTGQGEYYBRMX type=del size=1614215 2023-12-03T11:48:42.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.646+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.0 size=1048576 2023-12-03T11:48:42.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.647+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.1 size=1048576 2023-12-03T11:48:42.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.648+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~MUCPWC6PETY1.2 size=362916 2023-12-03T11:48:42.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.650+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=MUCPWC6PETY1 tag=7UM4ONEV9NHB type=write size=3508644 2023-12-03T11:48:42.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.652+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=SHAY8PY11NXN type=del size=735553 2023-12-03T11:48:42.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.653+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=3F8M9WNQ8AKZ type=del 2023-12-03T11:48:42.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.654+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=22U3SUTQ4672 type=write size=957518 2023-12-03T11:48:42.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.656+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=2M41K5HY4LP1 type=del 2023-12-03T11:48:42.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.657+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=EC2162UUXZX1 type=del size=2631603 2023-12-03T11:48:42.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.658+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.0 size=1048576 2023-12-03T11:48:42.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.659+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.1 size=1048576 2023-12-03T11:48:42.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.660+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~6KN0YQNOXZL2.2 size=666804 2023-12-03T11:48:42.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.663+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=6KN0YQNOXZL2 tag=63L43VYOKPW6 type=write size=3812532 2023-12-03T11:48:42.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.665+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~ZZT6HAGG4CQP.0 size=1048576 2023-12-03T11:48:42.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.666+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~ZZT6HAGG4CQP.1 size=510821 2023-12-03T11:48:42.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.666+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=ZZT6HAGG4CQP tag=ZDUYA4Y6W611 type=write size=2607973 2023-12-03T11:48:42.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.667+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~HRAKTYY68MC6.0 size=1048576 2023-12-03T11:48:42.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.667+0000 7f24574f5780 1 < canceled part key=_multipart_S0EH0Q0UA8HG.2~HRAKTYY68MC6.1 size=96921 2023-12-03T11:48:42.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.669+0000 7f24574f5780 1 < canceled multipart upload key=S0EH0Q0UA8HG upload=HRAKTYY68MC6 tag=A421JODFIEC5 type=write size=2194073 2023-12-03T11:48:42.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.671+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.0 size=1048576 2023-12-03T11:48:42.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.672+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.1 size=1048576 2023-12-03T11:48:42.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.673+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.2 size=277473 2023-12-03T11:48:42.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.673+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=6CIHUNQPSI3H tag=ER7I48ABELCX type=write size=3423201 2023-12-03T11:48:42.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.675+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=JYM3LB8CJ0XA type=del 2023-12-03T11:48:42.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.676+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=0DJN7ULSYGZH type=del size=477944 2023-12-03T11:48:42.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.677+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~LSQRIR5342AR.0 size=897961 2023-12-03T11:48:42.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.679+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=LSQRIR5342AR tag=VUDYEUOY7QPR type=write size=1946537 2023-12-03T11:48:42.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.680+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=1X0I8K46T9BO type=del size=2281633 2023-12-03T11:48:42.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.681+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.0 size=1048576 2023-12-03T11:48:42.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.682+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.1 size=1048576 2023-12-03T11:48:42.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.683+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~1TFQW21OZXWT.2 size=755741 2023-12-03T11:48:42.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.685+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=1TFQW21OZXWT tag=0X0CLBYZUHV4 type=write size=3901469 2023-12-03T11:48:42.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.686+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=W8NELTIRR09D type=del size=3229753 2023-12-03T11:48:42.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.688+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=5AS05XTO2E7M type=del 2023-12-03T11:48:42.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.689+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=3WVV6RXW3GJP type=del size=327604 2023-12-03T11:48:42.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.691+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=KHN53C61IEKM type=del size=1561113 2023-12-03T11:48:42.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.693+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=M429MESEZI7K type=del size=2830454 2023-12-03T11:48:42.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.694+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=HWS9F1GVINQ3 type=del 2023-12-03T11:48:42.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.695+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=13KV7KM0YQOV type=del size=328569 2023-12-03T11:48:42.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.696+0000 7f24574f5780 1 < canceled operation key=XZFHFS76WNR4 tag=0EGVQLL0Z8TK type=del size=2020363 2023-12-03T11:48:42.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.697+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=9E74NXZN45QS type=write size=668517 2023-12-03T11:48:42.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.698+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.0 size=1048576 2023-12-03T11:48:42.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.699+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.1 size=1048576 2023-12-03T11:48:42.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.700+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~7BM5QCSJKX1V.2 size=736081 2023-12-03T11:48:42.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.702+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=7BM5QCSJKX1V tag=61BZGUC0W7TK type=write size=3881809 2023-12-03T11:48:42.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.703+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=EFGGGUD9XOP7 type=del size=40794 2023-12-03T11:48:42.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.705+0000 7f24574f5780 1 < suggested update operation key=XO42YZR60KQE tag=W9MUSTDCNHW3 type=write size=629663 2023-12-03T11:48:42.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.706+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=IJY5X78N6BMY type=del size=1016349 2023-12-03T11:48:42.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.708+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~RLFJ0753370T.0 size=33190 2023-12-03T11:48:42.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.710+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=RLFJ0753370T tag=EWF86U302E8I type=write size=1081766 2023-12-03T11:48:42.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.712+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~GF3EQKA2TG0G.0 size=1048576 2023-12-03T11:48:42.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.715+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~GF3EQKA2TG0G.1 size=105808 2023-12-03T11:48:42.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.715+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=GF3EQKA2TG0G tag=GYFVP2C1JL0F type=write size=2202960 2023-12-03T11:48:42.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.717+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=UTGQGEYYBRMX type=del 2023-12-03T11:48:42.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.719+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.0 size=1048576 2023-12-03T11:48:42.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.720+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.1 size=1048576 2023-12-03T11:48:42.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.721+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.2 size=806306 2023-12-03T11:48:42.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.721+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=60HU4ZI2454L tag=DKRK8RTJM68N type=write size=3952034 2023-12-03T11:48:42.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.723+0000 7f24574f5780 1 < suggested remove operation key=MY3K9M4QRS3L tag=SHAY8PY11NXN type=del 2023-12-03T11:48:42.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.724+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=1B9NA8Z1JDDD type=del size=4158146 2023-12-03T11:48:42.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.725+0000 7f24574f5780 1 < completed write operation key=1V39QA10M1HY tag=22U3SUTQ4672 type=write size=957518 2023-12-03T11:48:42.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.726+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=ZZDAYIEI80DX type=del size=444879 2023-12-03T11:48:42.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.729+0000 7f24574f5780 1 < suggested update operation key=863EOZDYRWH3 tag=EC2162UUXZX1 type=del size=2631603 2023-12-03T11:48:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.730+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=JHZNO2RRSST4 type=write size=267566 2023-12-03T11:48:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.730+0000 7f24574f5780 1 < canceled part key=_multipart_WD7U31FW3NGZ.2~ZZT6HAGG4CQP.0 size=1048576 2023-12-03T11:48:42.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.731+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~ZZT6HAGG4CQP.1 size=510821 2023-12-03T11:48:42.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.734+0000 7f24574f5780 1 < canceled multipart upload key=WD7U31FW3NGZ upload=ZZT6HAGG4CQP tag=ZDUYA4Y6W611 type=write size=2607973 2023-12-03T11:48:42.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.735+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=FZMK24S5NJBE type=write size=408995 2023-12-03T11:48:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.736+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.0 size=1048576 2023-12-03T11:48:42.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.737+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.1 size=1048576 2023-12-03T11:48:42.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.738+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6CIHUNQPSI3H.2 size=277473 2023-12-03T11:48:42.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.740+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=6CIHUNQPSI3H tag=ER7I48ABELCX type=write size=3423201 2023-12-03T11:48:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.742+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~A0NVX56SNWNT.0 size=1045945 2023-12-03T11:48:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.742+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=A0NVX56SNWNT tag=93O85JOYFG8R type=write size=2094521 2023-12-03T11:48:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.743+0000 7f24574f5780 -1 < failed to complete operation key=S7Q5TCM499H1 tag=0DJN7ULSYGZH type=del size=477944: (22) Invalid argument 2023-12-03T11:48:42.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.744+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=10RGLUUM9ZCJ type=del size=4178358 2023-12-03T11:48:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.745+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=1X0I8K46T9BO type=del 2023-12-03T11:48:42.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.748+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~53UTRLWX7YWI.0 size=1048576 2023-12-03T11:48:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.749+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~53UTRLWX7YWI.1 size=911236 2023-12-03T11:48:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.749+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=53UTRLWX7YWI tag=KKM80CCEHKTW type=write size=3008388 2023-12-03T11:48:42.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.750+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=W8NELTIRR09D type=del 2023-12-03T11:48:42.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.752+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.0 size=1048576 2023-12-03T11:48:42.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.753+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.1 size=1048576 2023-12-03T11:48:42.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.754+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.2 size=98749 2023-12-03T11:48:42.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.754+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=NZTBCTA8P5HW tag=U9WMJ5WN87T5 type=write size=3244477 2023-12-03T11:48:42.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.755+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=3WVV6RXW3GJP type=del 2023-12-03T11:48:42.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.756+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=GBG506PB22S4 type=del size=2023098 2023-12-03T11:48:42.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.757+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=M429MESEZI7K type=del 2023-12-03T11:48:42.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.758+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=O579467CBSJE type=del size=262307 2023-12-03T11:48:42.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.759+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=13KV7KM0YQOV type=del size=328569 2023-12-03T11:48:42.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.760+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=WGAQ7NQFW1P4 type=del size=1582476 2023-12-03T11:48:42.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.761+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=9E74NXZN45QS type=write size=668517 2023-12-03T11:48:42.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.763+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=ODFOMCS3I1V5 type=del size=2075472 2023-12-03T11:48:42.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.764+0000 7f24574f5780 -1 < failed to complete operation key=863EOZDYRWH3 tag=EFGGGUD9XOP7 type=del size=40794: (22) Invalid argument 2023-12-03T11:48:42.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.766+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~922KSMMA3GS5.0 size=886194 2023-12-03T11:48:42.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.766+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=922KSMMA3GS5 tag=3HW6321S5Y0U type=write size=1934770 2023-12-03T11:48:42.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.767+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=IJY5X78N6BMY type=del 2023-12-03T11:48:42.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.768+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=W8UNB8R0VG33 type=del size=638072 2023-12-03T11:48:42.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.769+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~GF3EQKA2TG0G.0 size=1048576 2023-12-03T11:48:42.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.770+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~GF3EQKA2TG0G.1 size=105808 2023-12-03T11:48:42.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.772+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=GF3EQKA2TG0G tag=GYFVP2C1JL0F type=write size=2202960 2023-12-03T11:48:42.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.774+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=23FP999WX55A type=del size=2528299 2023-12-03T11:48:42.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.775+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.0 size=1048576 2023-12-03T11:48:42.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.776+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.1 size=1048576 2023-12-03T11:48:42.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.777+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~60HU4ZI2454L.2 size=806306 2023-12-03T11:48:42.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.779+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=60HU4ZI2454L tag=DKRK8RTJM68N type=write size=3952034 2023-12-03T11:48:42.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.781+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~U2K30YPNBPD4.0 size=1048576 2023-12-03T11:48:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.782+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~U2K30YPNBPD4.1 size=63762 2023-12-03T11:48:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.782+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=U2K30YPNBPD4 tag=91RBNYNTM478 type=write size=2160914 2023-12-03T11:48:42.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.783+0000 7f24574f5780 1 < canceled operation key=3EOEF9UMATIE tag=1B9NA8Z1JDDD type=del size=4158146 2023-12-03T11:48:42.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.784+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=1D0PRREDDU2O type=del size=3561563 2023-12-03T11:48:42.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.785+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=ZZDAYIEI80DX type=del 2023-12-03T11:48:42.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.787+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~PSSW1S7INO23.0 size=1048576 2023-12-03T11:48:42.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.788+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~PSSW1S7INO23.1 size=746937 2023-12-03T11:48:42.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.788+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=PSSW1S7INO23 tag=TRLV5Q0249W0 type=write size=2844089 2023-12-03T11:48:42.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.790+0000 7f24574f5780 1 < suggested remove operation key=S73M6UFMX5NH tag=JHZNO2RRSST4 type=write 2023-12-03T11:48:42.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.792+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~6F8DWYLFP8NT.0 size=1048576 2023-12-03T11:48:42.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.793+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~6F8DWYLFP8NT.1 size=933835 2023-12-03T11:48:42.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.793+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=6F8DWYLFP8NT tag=YMW2LBGTMHAM type=write size=3030987 2023-12-03T11:48:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.795+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=FZMK24S5NJBE type=write size=408995 2023-12-03T11:48:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.796+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=XYQYQK661AI8 type=del size=3348587 2023-12-03T11:48:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.797+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~A0NVX56SNWNT.0 size=1045945 2023-12-03T11:48:42.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.801+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=A0NVX56SNWNT tag=93O85JOYFG8R type=write size=2094521 2023-12-03T11:48:42.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.802+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=GGUIJT8PTOM3 type=del size=873625 2023-12-03T11:48:42.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.803+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=10RGLUUM9ZCJ type=del 2023-12-03T11:48:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.804+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=LQ3MPUDG402E type=del size=1985727 2023-12-03T11:48:42.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.805+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~53UTRLWX7YWI.0 size=1048576 2023-12-03T11:48:42.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.806+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~53UTRLWX7YWI.1 size=911236 2023-12-03T11:48:42.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.809+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=53UTRLWX7YWI tag=KKM80CCEHKTW type=write size=3008388 2023-12-03T11:48:42.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.811+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~PAOEEO0CZ1HH.0 size=1048576 2023-12-03T11:48:42.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.812+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~PAOEEO0CZ1HH.1 size=296140 2023-12-03T11:48:42.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.812+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=PAOEEO0CZ1HH tag=RYLV6NSSXIVT type=write size=2393292 2023-12-03T11:48:42.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.813+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.0 size=1048576 2023-12-03T11:48:42.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.814+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.1 size=1048576 2023-12-03T11:48:42.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.814+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~NZTBCTA8P5HW.2 size=98749 2023-12-03T11:48:42.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.816+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=NZTBCTA8P5HW tag=U9WMJ5WN87T5 type=write size=3244477 2023-12-03T11:48:42.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.818+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.0 size=1048576 2023-12-03T11:48:42.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.819+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.1 size=1048576 2023-12-03T11:48:42.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.820+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.2 size=896213 2023-12-03T11:48:42.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.820+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=V5I8A1DWGQ9G tag=TBGBL9GPMWT8 type=write size=4041941 2023-12-03T11:48:42.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.821+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=GBG506PB22S4 type=del 2023-12-03T11:48:42.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.822+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=BL9LRZP8BSP0 type=write size=600925 2023-12-03T11:48:42.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.825+0000 7f24574f5780 1 < suggested remove operation key=863EOZDYRWH3 tag=O579467CBSJE type=del 2023-12-03T11:48:42.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.826+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=7OZQ568JGUDM type=del size=837236 2023-12-03T11:48:42.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.827+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=WGAQ7NQFW1P4 type=del 2023-12-03T11:48:42.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.829+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.0 size=1048576 2023-12-03T11:48:42.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.830+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.1 size=1048576 2023-12-03T11:48:42.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.831+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.2 size=930464 2023-12-03T11:48:42.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.831+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=4VF7AAPYLIQ9 tag=QT88CQ981TH3 type=write size=4076192 2023-12-03T11:48:42.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.832+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=ODFOMCS3I1V5 type=del 2023-12-03T11:48:42.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.833+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=3O32LHIQHL5Y type=del size=3321663 2023-12-03T11:48:42.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.834+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~922KSMMA3GS5.0 size=886194 2023-12-03T11:48:42.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.836+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=922KSMMA3GS5 tag=3HW6321S5Y0U type=write size=1934770 2023-12-03T11:48:42.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.837+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=SZTSVE2A105N type=del size=952397 2023-12-03T11:48:42.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.838+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=W8UNB8R0VG33 type=del 2023-12-03T11:48:42.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.840+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=W1UJA7EGSIHS type=del size=3051937 2023-12-03T11:48:42.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.842+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=23FP999WX55A type=del 2023-12-03T11:48:42.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.845+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=V8KV7LS5RAU0 type=del size=2472340 2023-12-03T11:48:42.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.846+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~U2K30YPNBPD4.0 size=1048576 2023-12-03T11:48:42.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.847+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~U2K30YPNBPD4.1 size=63762 2023-12-03T11:48:42.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.849+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=U2K30YPNBPD4 tag=91RBNYNTM478 type=write size=2160914 2023-12-03T11:48:42.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.850+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=PGV584DQ7D3R type=del size=178932 2023-12-03T11:48:42.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.851+0000 7f24574f5780 -1 < failed to complete operation key=863EOZDYRWH3 tag=1D0PRREDDU2O type=del size=3561563: (22) Invalid argument 2023-12-03T11:48:42.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.852+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=QS208SL835NF type=del size=2231969 2023-12-03T11:48:42.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.853+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~PSSW1S7INO23.0 size=1048576 2023-12-03T11:48:42.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.854+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~PSSW1S7INO23.1 size=746937 2023-12-03T11:48:42.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.856+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=PSSW1S7INO23 tag=TRLV5Q0249W0 type=write size=2844089 2023-12-03T11:48:42.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.857+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=08XD1J8UH9WG type=del size=2741981 2023-12-03T11:48:42.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.858+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~6F8DWYLFP8NT.0 size=1048576 2023-12-03T11:48:42.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.859+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~6F8DWYLFP8NT.1 size=933835 2023-12-03T11:48:42.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.861+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=6F8DWYLFP8NT tag=YMW2LBGTMHAM type=write size=3030987 2023-12-03T11:48:42.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.862+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=MK8TM0NWASDD type=del size=949954 2023-12-03T11:48:42.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.865+0000 7f24574f5780 1 < suggested update operation key=863EOZDYRWH3 tag=XYQYQK661AI8 type=del size=3348587 2023-12-03T11:48:42.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.866+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=2P41862PKIT6 type=del size=3447014 2023-12-03T11:48:42.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.867+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=GGUIJT8PTOM3 type=del 2023-12-03T11:48:42.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.868+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=JWYG6H9USMD2 type=write size=422409 2023-12-03T11:48:42.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.869+0000 7f24574f5780 -1 < failed to complete operation key=863EOZDYRWH3 tag=LQ3MPUDG402E type=del size=1985727: (22) Invalid argument 2023-12-03T11:48:42.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.871+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.0 size=1048576 2023-12-03T11:48:42.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.872+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.1 size=1048576 2023-12-03T11:48:42.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.873+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.2 size=492963 2023-12-03T11:48:42.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.873+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=JPTU8GENTCWJ tag=I4ZAO78YSS8R type=write size=3638691 2023-12-03T11:48:42.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.874+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~PAOEEO0CZ1HH.0 size=1048576 2023-12-03T11:48:42.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.875+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~PAOEEO0CZ1HH.1 size=296140 2023-12-03T11:48:42.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.877+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=PAOEEO0CZ1HH tag=RYLV6NSSXIVT type=write size=2393292 2023-12-03T11:48:42.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.879+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.0 size=1048576 2023-12-03T11:48:42.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.880+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.1 size=1048576 2023-12-03T11:48:42.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.881+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.2 size=94092 2023-12-03T11:48:42.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.881+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=PPXK4DI31ASH tag=LSA8CGAYDI3Z type=write size=3239820 2023-12-03T11:48:42.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.882+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.0 size=1048576 2023-12-03T11:48:42.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.883+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.1 size=1048576 2023-12-03T11:48:42.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.884+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~V5I8A1DWGQ9G.2 size=896213 2023-12-03T11:48:42.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.886+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=V5I8A1DWGQ9G tag=TBGBL9GPMWT8 type=write size=4041941 2023-12-03T11:48:42.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.887+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=1DLUT0986TQY type=del size=2823035 2023-12-03T11:48:42.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.888+0000 7f24574f5780 1 < canceled operation key=XRHVYVADRWET tag=BL9LRZP8BSP0 type=write size=600925 2023-12-03T11:48:42.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.890+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=JB6SKKPGULZS type=del size=564582 2023-12-03T11:48:42.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.891+0000 7f24574f5780 -1 < failed to complete operation key=863EOZDYRWH3 tag=7OZQ568JGUDM type=del size=837236: (22) Invalid argument 2023-12-03T11:48:42.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.892+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=W73LAL7X29PP type=del size=1404046 2023-12-03T11:48:42.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.893+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.0 size=1048576 2023-12-03T11:48:42.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.894+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.1 size=1048576 2023-12-03T11:48:42.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.895+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~4VF7AAPYLIQ9.2 size=930464 2023-12-03T11:48:42.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.897+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=4VF7AAPYLIQ9 tag=QT88CQ981TH3 type=write size=4076192 2023-12-03T11:48:42.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.899+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~SLCRZWK0E9YK.0 size=496882 2023-12-03T11:48:42.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.899+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=SLCRZWK0E9YK tag=9UHPGEQHJAMK type=write size=1545458 2023-12-03T11:48:42.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.900+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=3O32LHIQHL5Y type=del 2023-12-03T11:48:42.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.901+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=RSG7G9DTUHFZ type=del size=1494274 2023-12-03T11:48:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.903+0000 7f24574f5780 1 < canceled operation key=XO42YZR60KQE tag=SZTSVE2A105N type=del size=952397 2023-12-03T11:48:42.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.905+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~UB2PSPMZ4JDY.0 size=619854 2023-12-03T11:48:42.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.905+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=UB2PSPMZ4JDY tag=HL3O0BK9IKLW type=write size=1668430 2023-12-03T11:48:42.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.906+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=W1UJA7EGSIHS type=del 2023-12-03T11:48:42.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.907+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=V0SREQL4S7Q8 type=del size=959843 2023-12-03T11:48:42.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.908+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=V8KV7LS5RAU0 type=del 2023-12-03T11:48:42.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.910+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=IZETV1TK8WGE type=del size=2864438 2023-12-03T11:48:42.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.911+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=PGV584DQ7D3R type=del 2023-12-03T11:48:42.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.913+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~I5W9W30ZOY8B.0 size=1048576 2023-12-03T11:48:42.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.914+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~I5W9W30ZOY8B.1 size=358340 2023-12-03T11:48:42.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.914+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=I5W9W30ZOY8B tag=NSV7W1MMK08Y type=write size=2455492 2023-12-03T11:48:42.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.916+0000 7f24574f5780 1 < suggested update operation key=XO42YZR60KQE tag=QS208SL835NF type=del size=2231969 2023-12-03T11:48:42.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.918+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~80ESZJBKGCCF.0 size=1048576 2023-12-03T11:48:42.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.919+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~80ESZJBKGCCF.1 size=116644 2023-12-03T11:48:42.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.919+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=80ESZJBKGCCF tag=KD6KCB7TPP27 type=write size=2213796 2023-12-03T11:48:42.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.920+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=08XD1J8UH9WG type=del 2023-12-03T11:48:42.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.921+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=SKJ0TNYNYQ7H type=del size=1421456 2023-12-03T11:48:42.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.922+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=MK8TM0NWASDD type=del 2023-12-03T11:48:42.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.923+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=5L23KIQW7NDC type=del size=737163 2023-12-03T11:48:42.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.926+0000 7f24574f5780 1 < suggested update operation key=863EOZDYRWH3 tag=2P41862PKIT6 type=del size=3447014 2023-12-03T11:48:42.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.928+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~KMM039P1U25D.0 size=1048576 2023-12-03T11:48:42.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.929+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~KMM039P1U25D.1 size=838359 2023-12-03T11:48:42.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.929+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=KMM039P1U25D tag=R326XU8NEGLB type=write size=2935511 2023-12-03T11:48:42.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.930+0000 7f24574f5780 1 < completed write operation key=S0EH0Q0UA8HG tag=JWYG6H9USMD2 type=write size=422409 2023-12-03T11:48:42.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.932+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~V77LJFPCFDQH.0 size=1048576 2023-12-03T11:48:42.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.933+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~V77LJFPCFDQH.1 size=560984 2023-12-03T11:48:42.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.933+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=V77LJFPCFDQH tag=Y8ANEZNGDEXG type=write size=2658136 2023-12-03T11:48:42.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.934+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.0 size=1048576 2023-12-03T11:48:42.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.935+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.1 size=1048576 2023-12-03T11:48:42.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.936+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~JPTU8GENTCWJ.2 size=492963 2023-12-03T11:48:42.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.938+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=JPTU8GENTCWJ tag=I4ZAO78YSS8R type=write size=3638691 2023-12-03T11:48:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.939+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=937F8X1RCORU type=del size=1376514 2023-12-03T11:48:42.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.939+0000 7f24574f5780 1 < canceled part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.0 size=1048576 2023-12-03T11:48:42.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.940+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.1 size=1048576 2023-12-03T11:48:42.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.941+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~PPXK4DI31ASH.2 size=94092 2023-12-03T11:48:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.943+0000 7f24574f5780 1 < canceled multipart upload key=863EOZDYRWH3 upload=PPXK4DI31ASH tag=LSA8CGAYDI3Z type=write size=3239820 2023-12-03T11:48:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.944+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=U9VXYEIX8695 type=write size=600923 2023-12-03T11:48:42.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.946+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=1DLUT0986TQY type=del 2023-12-03T11:48:42.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.948+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=76SLR2EA8WCJ type=del size=472006 2023-12-03T11:48:42.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.949+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=JB6SKKPGULZS type=del 2023-12-03T11:48:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.952+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~12AATNICFT8Q.0 size=1048576 2023-12-03T11:48:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.953+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~12AATNICFT8Q.1 size=956415 2023-12-03T11:48:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.953+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=12AATNICFT8Q tag=RJA96MT14XM9 type=write size=3053567 2023-12-03T11:48:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.954+0000 7f24574f5780 1 < canceled operation key=XRHVYVADRWET tag=W73LAL7X29PP type=del size=1404046 2023-12-03T11:48:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.955+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=B3KHRXKPCPO3 type=del size=369127 2023-12-03T11:48:42.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.956+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~SLCRZWK0E9YK.0 size=496882 2023-12-03T11:48:42.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.958+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=SLCRZWK0E9YK tag=9UHPGEQHJAMK type=write size=1545458 2023-12-03T11:48:42.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.959+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=5Q1FF2NRCUGE type=del size=987147 2023-12-03T11:48:42.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.960+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=RSG7G9DTUHFZ type=del 2023-12-03T11:48:42.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.962+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.0 size=1048576 2023-12-03T11:48:42.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.963+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.1 size=1048576 2023-12-03T11:48:42.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.964+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.2 size=425329 2023-12-03T11:48:42.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.964+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=VWZ2C0WIBAZJ tag=TPR6IFK5R553 type=write size=3571057 2023-12-03T11:48:42.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.965+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~UB2PSPMZ4JDY.0 size=619854 2023-12-03T11:48:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.967+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=UB2PSPMZ4JDY tag=HL3O0BK9IKLW type=write size=1668430 2023-12-03T11:48:42.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.969+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~TNXG7YMTW636.0 size=1048576 2023-12-03T11:48:42.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.970+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~TNXG7YMTW636.1 size=914743 2023-12-03T11:48:42.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.970+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=TNXG7YMTW636 tag=91L8700HQRKE type=write size=3011895 2023-12-03T11:48:42.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.971+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=V0SREQL4S7Q8 type=del 2023-12-03T11:48:42.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.972+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=H2AIE8BSUFCS type=del size=2292897 2023-12-03T11:48:42.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.973+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=IZETV1TK8WGE type=del 2023-12-03T11:48:42.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.976+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.0 size=1048576 2023-12-03T11:48:42.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.976+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.1 size=1048576 2023-12-03T11:48:42.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.977+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.2 size=345380 2023-12-03T11:48:42.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.977+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=UZBCLOJMW7SE tag=KZHAIX9R4YMT type=write size=3491108 2023-12-03T11:48:42.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.979+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~I5W9W30ZOY8B.0 size=1048576 2023-12-03T11:48:42.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.980+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~I5W9W30ZOY8B.1 size=358340 2023-12-03T11:48:42.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.981+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=I5W9W30ZOY8B tag=NSV7W1MMK08Y type=write size=2455492 2023-12-03T11:48:42.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.984+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5VI1COFQOP99.0 size=1048576 2023-12-03T11:48:42.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.984+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5VI1COFQOP99.1 size=222205 2023-12-03T11:48:42.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.984+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=5VI1COFQOP99 tag=S3ZFEQCSWQEC type=write size=2319357 2023-12-03T11:48:42.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.986+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~80ESZJBKGCCF.0 size=1048576 2023-12-03T11:48:42.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.987+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~80ESZJBKGCCF.1 size=116644 2023-12-03T11:48:42.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.989+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=80ESZJBKGCCF tag=KD6KCB7TPP27 type=write size=2213796 2023-12-03T11:48:42.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.990+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=IL495AQWXF98 type=del size=1584199 2023-12-03T11:48:42.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.991+0000 7f24574f5780 -1 < failed to complete operation key=ON2E6H35N9FJ tag=SKJ0TNYNYQ7H type=del size=1421456: (22) Invalid argument 2023-12-03T11:48:42.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.992+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=JR7153I2VCDM type=write size=390075 2023-12-03T11:48:42.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.993+0000 7f24574f5780 1 < canceled operation key=PGTGVLK4DBN0 tag=5L23KIQW7NDC type=del size=737163 2023-12-03T11:48:42.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.995+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=6TV80B2UMMKY type=del size=3510672 2023-12-03T11:48:42.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.996+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~KMM039P1U25D.0 size=1048576 2023-12-03T11:48:42.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.997+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~KMM039P1U25D.1 size=838359 2023-12-03T11:48:43.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:42.999+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=KMM039P1U25D tag=R326XU8NEGLB type=write size=2935511 2023-12-03T11:48:43.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.000+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=AFEX6WRD1VKN type=del size=2286952 2023-12-03T11:48:43.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.002+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~V77LJFPCFDQH.0 size=1048576 2023-12-03T11:48:43.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.003+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~V77LJFPCFDQH.1 size=560984 2023-12-03T11:48:43.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.005+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=V77LJFPCFDQH tag=Y8ANEZNGDEXG type=write size=2658136 2023-12-03T11:48:43.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.006+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=IBMPWTNS5B9T type=del size=1666519 2023-12-03T11:48:43.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.007+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=937F8X1RCORU type=del 2023-12-03T11:48:43.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.008+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=L8ELU96CL7SS type=del size=974916 2023-12-03T11:48:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.009+0000 7f24574f5780 1 < completed write operation key=U043YAPGUBT5 tag=U9VXYEIX8695 type=write size=600923 2023-12-03T11:48:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.011+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.0 size=1048576 2023-12-03T11:48:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.012+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.1 size=1048576 2023-12-03T11:48:43.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.013+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.2 size=549532 2023-12-03T11:48:43.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.013+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=L42XQ7OGCT60 tag=HEOVRKNMCGT2 type=write size=3695260 2023-12-03T11:48:43.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.014+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=76SLR2EA8WCJ type=del 2023-12-03T11:48:43.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.015+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=JHIIKZ8BTHNZ type=del size=4087411 2023-12-03T11:48:43.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.016+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~12AATNICFT8Q.0 size=1048576 2023-12-03T11:48:43.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.017+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~12AATNICFT8Q.1 size=956415 2023-12-03T11:48:43.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.019+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=12AATNICFT8Q tag=RJA96MT14XM9 type=write size=3053567 2023-12-03T11:48:43.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.021+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=MNG6WWB9UN64 type=del size=883859 2023-12-03T11:48:43.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.022+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=B3KHRXKPCPO3 type=del 2023-12-03T11:48:43.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.023+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=DHQSXA085Y1M type=del size=1957202 2023-12-03T11:48:43.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.024+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=5Q1FF2NRCUGE type=del 2023-12-03T11:48:43.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.025+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=I3L0BBIFPIJ9 type=del size=1917494 2023-12-03T11:48:43.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.026+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.0 size=1048576 2023-12-03T11:48:43.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.027+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.1 size=1048576 2023-12-03T11:48:43.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.029+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~VWZ2C0WIBAZJ.2 size=425329 2023-12-03T11:48:43.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.031+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=VWZ2C0WIBAZJ tag=TPR6IFK5R553 type=write size=3571057 2023-12-03T11:48:43.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.032+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=EV94U0H2E1X2 type=write size=826694 2023-12-03T11:48:43.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.033+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~TNXG7YMTW636.0 size=1048576 2023-12-03T11:48:43.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.034+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~TNXG7YMTW636.1 size=914743 2023-12-03T11:48:43.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.036+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=TNXG7YMTW636 tag=91L8700HQRKE type=write size=3011895 2023-12-03T11:48:43.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.038+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~7XGPF382R1LR.0 size=1048576 2023-12-03T11:48:43.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.039+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~7XGPF382R1LR.1 size=985967 2023-12-03T11:48:43.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.039+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=7XGPF382R1LR tag=OOL5EGU3JWUT type=write size=3083119 2023-12-03T11:48:43.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.041+0000 7f24574f5780 1 < suggested remove operation key=863EOZDYRWH3 tag=H2AIE8BSUFCS type=del 2023-12-03T11:48:43.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.042+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=HQJ6P8U6RFC6 type=write size=788921 2023-12-03T11:48:43.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.044+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.0 size=1048576 2023-12-03T11:48:43.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.046+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.1 size=1048576 2023-12-03T11:48:43.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.047+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~UZBCLOJMW7SE.2 size=345380 2023-12-03T11:48:43.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.049+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=UZBCLOJMW7SE tag=KZHAIX9R4YMT type=write size=3491108 2023-12-03T11:48:43.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.051+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=7S3Q9B9J4SNZ type=del size=1515201 2023-12-03T11:48:43.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.052+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5VI1COFQOP99.0 size=1048576 2023-12-03T11:48:43.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.053+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5VI1COFQOP99.1 size=222205 2023-12-03T11:48:43.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.055+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=5VI1COFQOP99 tag=S3ZFEQCSWQEC type=write size=2319357 2023-12-03T11:48:43.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.057+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=UADD07Q8B0VK type=write size=577347 2023-12-03T11:48:43.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.058+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=IL495AQWXF98 type=del 2023-12-03T11:48:43.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.061+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~MCBZ6K93NSXQ.0 size=1048576 2023-12-03T11:48:43.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.062+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~MCBZ6K93NSXQ.1 size=783736 2023-12-03T11:48:43.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.062+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=MCBZ6K93NSXQ tag=97G89JOIDU4F type=write size=2880888 2023-12-03T11:48:43.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.064+0000 7f24574f5780 1 < completed write operation key=XO42YZR60KQE tag=JR7153I2VCDM type=write size=390075 2023-12-03T11:48:43.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.065+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=2OSRS8UXFZGH type=write size=945115 2023-12-03T11:48:43.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.066+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=6TV80B2UMMKY type=del 2023-12-03T11:48:43.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.068+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~4QZN2QZNM6II.0 size=1048576 2023-12-03T11:48:43.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.069+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~4QZN2QZNM6II.1 size=394658 2023-12-03T11:48:43.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.069+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=4QZN2QZNM6II tag=U28BRQL834X5 type=write size=2491810 2023-12-03T11:48:43.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.070+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=AFEX6WRD1VKN type=del 2023-12-03T11:48:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.072+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~TOIQGJCXNZS2.0 size=84365 2023-12-03T11:48:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.072+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=TOIQGJCXNZS2 tag=J6VOKJY9UXT6 type=write size=1132941 2023-12-03T11:48:43.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.073+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=IBMPWTNS5B9T type=del 2023-12-03T11:48:43.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.074+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=AQ3KOBXT456Y type=write size=522328 2023-12-03T11:48:43.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.075+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=L8ELU96CL7SS type=del 2023-12-03T11:48:43.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.078+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.0 size=1048576 2023-12-03T11:48:43.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.079+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.1 size=1048576 2023-12-03T11:48:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.080+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.2 size=849942 2023-12-03T11:48:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.080+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=OWUI7EMFHA70 tag=R0EU7ULU72L1 type=write size=3995670 2023-12-03T11:48:43.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.081+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.0 size=1048576 2023-12-03T11:48:43.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.082+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.1 size=1048576 2023-12-03T11:48:43.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.083+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~L42XQ7OGCT60.2 size=549532 2023-12-03T11:48:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.085+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=L42XQ7OGCT60 tag=HEOVRKNMCGT2 type=write size=3695260 2023-12-03T11:48:43.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.087+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.0 size=1048576 2023-12-03T11:48:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.088+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.1 size=1048576 2023-12-03T11:48:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.089+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.2 size=520182 2023-12-03T11:48:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.089+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=7OT0ZM28CQZ6 tag=QPY2HH8VKBL4 type=write size=3665910 2023-12-03T11:48:43.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.090+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=JHIIKZ8BTHNZ type=del 2023-12-03T11:48:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.092+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=S3WXSVE0P7QY type=write size=796164 2023-12-03T11:48:43.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.093+0000 7f24574f5780 1 < canceled operation key=9LKFY1U0JRBQ tag=MNG6WWB9UN64 type=del size=883859 2023-12-03T11:48:43.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.095+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~Q7O93IM5DGCT.0 size=1048576 2023-12-03T11:48:43.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.096+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~Q7O93IM5DGCT.1 size=336504 2023-12-03T11:48:43.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.096+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=Q7O93IM5DGCT tag=5LP9595YOXNA type=write size=2433656 2023-12-03T11:48:43.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.097+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=DHQSXA085Y1M type=del 2023-12-03T11:48:43.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.098+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=LZX233XK5XFN type=write size=128792 2023-12-03T11:48:43.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.099+0000 7f24574f5780 1 < canceled operation key=XO42YZR60KQE tag=I3L0BBIFPIJ9 type=del size=1917494 2023-12-03T11:48:43.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.100+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=F3EJB8O9Q5YB type=del size=4089377 2023-12-03T11:48:43.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.102+0000 7f24574f5780 1 < completed write operation key=WZMWCSAZX8RB tag=EV94U0H2E1X2 type=write size=826694 2023-12-03T11:48:43.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.103+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=7ANRYXMEH2AH type=del size=3162158 2023-12-03T11:48:43.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.104+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~7XGPF382R1LR.0 size=1048576 2023-12-03T11:48:43.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.105+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~7XGPF382R1LR.1 size=985967 2023-12-03T11:48:43.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.107+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=7XGPF382R1LR tag=OOL5EGU3JWUT type=write size=3083119 2023-12-03T11:48:43.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.108+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=BEVLWBF9M22H type=del size=149399 2023-12-03T11:48:43.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.109+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=HQJ6P8U6RFC6 type=write size=788921 2023-12-03T11:48:43.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.111+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~HZHKNTIENT4H.0 size=707461 2023-12-03T11:48:43.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.111+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=HZHKNTIENT4H tag=22L2HX9AQGZ1 type=write size=1756037 2023-12-03T11:48:43.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.112+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=7S3Q9B9J4SNZ type=del 2023-12-03T11:48:43.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.114+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~M1V4R82ZP62G.0 size=1048576 2023-12-03T11:48:43.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.115+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~M1V4R82ZP62G.1 size=212949 2023-12-03T11:48:43.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.115+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=M1V4R82ZP62G tag=XNCXHIPW2UPN type=write size=2310101 2023-12-03T11:48:43.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.117+0000 7f24574f5780 1 < completed write operation key=9LKFY1U0JRBQ tag=UADD07Q8B0VK type=write size=577347 2023-12-03T11:48:43.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.118+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=021ZUHN23RNY type=del size=2677523 2023-12-03T11:48:43.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.119+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~MCBZ6K93NSXQ.0 size=1048576 2023-12-03T11:48:43.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.120+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~MCBZ6K93NSXQ.1 size=783736 2023-12-03T11:48:43.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.122+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=MCBZ6K93NSXQ tag=97G89JOIDU4F type=write size=2880888 2023-12-03T11:48:43.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.124+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~JYZUOTUD0K1O.0 size=844540 2023-12-03T11:48:43.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.124+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=JYZUOTUD0K1O tag=9OZ0FC3HGFOJ type=write size=1893116 2023-12-03T11:48:43.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.125+0000 7f24574f5780 1 < canceled operation key=QVLPCWC4T8N8 tag=2OSRS8UXFZGH type=write size=945115 2023-12-03T11:48:43.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.126+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=7N4J4WY639Z5 type=del size=1812575 2023-12-03T11:48:43.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.127+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~4QZN2QZNM6II.0 size=1048576 2023-12-03T11:48:43.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.128+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~4QZN2QZNM6II.1 size=394658 2023-12-03T11:48:43.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.130+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=4QZN2QZNM6II tag=U28BRQL834X5 type=write size=2491810 2023-12-03T11:48:43.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.131+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=OB4GKFJCQJN7 type=del size=3274943 2023-12-03T11:48:43.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.132+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~TOIQGJCXNZS2.0 size=84365 2023-12-03T11:48:43.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.134+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=TOIQGJCXNZS2 tag=J6VOKJY9UXT6 type=write size=1132941 2023-12-03T11:48:43.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.136+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.0 size=1048576 2023-12-03T11:48:43.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.137+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.1 size=1048576 2023-12-03T11:48:43.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.138+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.2 size=423188 2023-12-03T11:48:43.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.138+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=ELEQLOKX9KGI tag=LM0G75WCX3UG type=write size=3568916 2023-12-03T11:48:43.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.139+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=AQ3KOBXT456Y type=write size=522328 2023-12-03T11:48:43.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.142+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~MNL0G5C6M5MG.0 size=489210 2023-12-03T11:48:43.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.142+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=MNL0G5C6M5MG tag=3CN91YTTRV2Q type=write size=1537786 2023-12-03T11:48:43.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.142+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.0 size=1048576 2023-12-03T11:48:43.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.143+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.1 size=1048576 2023-12-03T11:48:43.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.144+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~OWUI7EMFHA70.2 size=849942 2023-12-03T11:48:43.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.146+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=OWUI7EMFHA70 tag=R0EU7ULU72L1 type=write size=3995670 2023-12-03T11:48:43.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.148+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~DGK1EY2C1282.0 size=1048576 2023-12-03T11:48:43.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.148+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~DGK1EY2C1282.1 size=379770 2023-12-03T11:48:43.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.148+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=DGK1EY2C1282 tag=FKHZMA0LRTQO type=write size=2476922 2023-12-03T11:48:43.151 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.150+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.0 size=1048576 2023-12-03T11:48:43.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.151+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.1 size=1048576 2023-12-03T11:48:43.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.152+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~7OT0ZM28CQZ6.2 size=520182 2023-12-03T11:48:43.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.153+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=7OT0ZM28CQZ6 tag=QPY2HH8VKBL4 type=write size=3665910 2023-12-03T11:48:43.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.156+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~1TT8K5V4UIWU.0 size=957193 2023-12-03T11:48:43.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.156+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=1TT8K5V4UIWU tag=NMC87QVSXVPR type=write size=2005769 2023-12-03T11:48:43.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.157+0000 7f24574f5780 1 < completed write operation key=S7Q5TCM499H1 tag=S3WXSVE0P7QY type=write size=796164 2023-12-03T11:48:43.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.159+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~PZKA3QW72DBS.0 size=879923 2023-12-03T11:48:43.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.159+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=PZKA3QW72DBS tag=MTSB2ER03YT6 type=write size=1928499 2023-12-03T11:48:43.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.160+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~Q7O93IM5DGCT.0 size=1048576 2023-12-03T11:48:43.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.161+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~Q7O93IM5DGCT.1 size=336504 2023-12-03T11:48:43.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.163+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=Q7O93IM5DGCT tag=5LP9595YOXNA type=write size=2433656 2023-12-03T11:48:43.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.165+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=KIEI1Z2H8X9U type=del size=1466007 2023-12-03T11:48:43.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.166+0000 7f24574f5780 1 < completed write operation key=BV8C0VSJ1GTW tag=LZX233XK5XFN type=write size=128792 2023-12-03T11:48:43.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.169+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~7K8OBP9EK0Y4.0 size=1048576 2023-12-03T11:48:43.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.172+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~7K8OBP9EK0Y4.1 size=722791 2023-12-03T11:48:43.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.172+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=7K8OBP9EK0Y4 tag=KCFM4FV4D442 type=write size=2819943 2023-12-03T11:48:43.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.174+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=F3EJB8O9Q5YB type=del 2023-12-03T11:48:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.176+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=OGKHLGRK2JEW type=del size=3227985 2023-12-03T11:48:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.177+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=7ANRYXMEH2AH type=del 2023-12-03T11:48:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.178+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=JA0NT4MT15RP type=del size=376784 2023-12-03T11:48:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.179+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=BEVLWBF9M22H type=del 2023-12-03T11:48:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.181+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~ZB7AESMPBYN9.0 size=298009 2023-12-03T11:48:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.181+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=ZB7AESMPBYN9 tag=RLBJKMYS51TF type=write size=1346585 2023-12-03T11:48:43.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.183+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~HZHKNTIENT4H.0 size=707461 2023-12-03T11:48:43.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.184+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=HZHKNTIENT4H tag=22L2HX9AQGZ1 type=write size=1756037 2023-12-03T11:48:43.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.186+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=HYWELT7GH4LX type=del size=3050451 2023-12-03T11:48:43.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.187+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~M1V4R82ZP62G.0 size=1048576 2023-12-03T11:48:43.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.188+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~M1V4R82ZP62G.1 size=212949 2023-12-03T11:48:43.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.190+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=M1V4R82ZP62G tag=XNCXHIPW2UPN type=write size=2310101 2023-12-03T11:48:43.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.192+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~PZRMPBOQ0SN1.0 size=179332 2023-12-03T11:48:43.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.192+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=PZRMPBOQ0SN1 tag=1GES16JI79GB type=write size=1227908 2023-12-03T11:48:43.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.193+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=021ZUHN23RNY type=del 2023-12-03T11:48:43.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.194+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=1O0MTXAYLSV1 type=del size=3657292 2023-12-03T11:48:43.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.195+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~JYZUOTUD0K1O.0 size=844540 2023-12-03T11:48:43.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.197+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=JYZUOTUD0K1O tag=9OZ0FC3HGFOJ type=write size=1893116 2023-12-03T11:48:43.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.198+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=JGZJEP62C6CF type=del size=858450 2023-12-03T11:48:43.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.200+0000 7f24574f5780 1 < suggested update operation key=863EOZDYRWH3 tag=7N4J4WY639Z5 type=del size=1812575 2023-12-03T11:48:43.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.202+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.0 size=1048576 2023-12-03T11:48:43.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.203+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.1 size=1048576 2023-12-03T11:48:43.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.204+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.2 size=306894 2023-12-03T11:48:43.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.204+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=DUF4C7MMML9A tag=KZEMSL88ZP2G type=write size=3452622 2023-12-03T11:48:43.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.205+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=OB4GKFJCQJN7 type=del 2023-12-03T11:48:43.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.206+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=7RCDUC3YYY00 type=write size=480622 2023-12-03T11:48:43.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.207+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.0 size=1048576 2023-12-03T11:48:43.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.208+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.1 size=1048576 2023-12-03T11:48:43.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.209+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~ELEQLOKX9KGI.2 size=423188 2023-12-03T11:48:43.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.211+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=ELEQLOKX9KGI tag=LM0G75WCX3UG type=write size=3568916 2023-12-03T11:48:43.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.213+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~5BZE6TLUB15Q.0 size=1048576 2023-12-03T11:48:43.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.214+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~5BZE6TLUB15Q.1 size=391205 2023-12-03T11:48:43.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.214+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=5BZE6TLUB15Q tag=IVF13LMNY0ME type=write size=2488357 2023-12-03T11:48:43.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.215+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~MNL0G5C6M5MG.0 size=489210 2023-12-03T11:48:43.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.217+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=MNL0G5C6M5MG tag=3CN91YTTRV2Q type=write size=1537786 2023-12-03T11:48:43.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.218+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=LD16JO0R55KK type=del size=3981222 2023-12-03T11:48:43.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.219+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~DGK1EY2C1282.0 size=1048576 2023-12-03T11:48:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.220+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~DGK1EY2C1282.1 size=379770 2023-12-03T11:48:43.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.222+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=DGK1EY2C1282 tag=FKHZMA0LRTQO type=write size=2476922 2023-12-03T11:48:43.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.223+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=Q45LT80SIM53 type=del size=3508226 2023-12-03T11:48:43.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.224+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~1TT8K5V4UIWU.0 size=957193 2023-12-03T11:48:43.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.226+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=1TT8K5V4UIWU tag=NMC87QVSXVPR type=write size=2005769 2023-12-03T11:48:43.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.228+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.0 size=1048576 2023-12-03T11:48:43.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.229+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.1 size=1048576 2023-12-03T11:48:43.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.230+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.2 size=165707 2023-12-03T11:48:43.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.230+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=7ZEKHUW9HRP1 tag=9C1S8WIDVO5I type=write size=3311435 2023-12-03T11:48:43.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.231+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~PZKA3QW72DBS.0 size=879923 2023-12-03T11:48:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.233+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=PZKA3QW72DBS tag=MTSB2ER03YT6 type=write size=1928499 2023-12-03T11:48:43.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.235+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~I46YWX5UFIBI.0 size=572844 2023-12-03T11:48:43.237 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.235+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=I46YWX5UFIBI tag=IOWQ24NOBW9I type=write size=1621420 2023-12-03T11:48:43.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.237+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=KIEI1Z2H8X9U type=del 2023-12-03T11:48:43.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.238+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=TMLEJ7DAL5T3 type=del size=3219126 2023-12-03T11:48:43.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.239+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~7K8OBP9EK0Y4.0 size=1048576 2023-12-03T11:48:43.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.240+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~7K8OBP9EK0Y4.1 size=722791 2023-12-03T11:48:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.242+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=7K8OBP9EK0Y4 tag=KCFM4FV4D442 type=write size=2819943 2023-12-03T11:48:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.243+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=VTAB0X2L5T1R type=write size=91677 2023-12-03T11:48:43.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.244+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=OGKHLGRK2JEW type=del 2023-12-03T11:48:43.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.246+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=XKGU53NN1OPA type=del size=2303857 2023-12-03T11:48:43.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.247+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=JA0NT4MT15RP type=del 2023-12-03T11:48:43.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.249+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~8P5QPK37LNDP.0 size=1048576 2023-12-03T11:48:43.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.250+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~8P5QPK37LNDP.1 size=651546 2023-12-03T11:48:43.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.250+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=8P5QPK37LNDP tag=5QJNHGUL0FEH type=write size=2748698 2023-12-03T11:48:43.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.251+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~ZB7AESMPBYN9.0 size=298009 2023-12-03T11:48:43.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.253+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=ZB7AESMPBYN9 tag=RLBJKMYS51TF type=write size=1346585 2023-12-03T11:48:43.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.254+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=L3YP6ZVC9QW1 type=del size=2861132 2023-12-03T11:48:43.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.256+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=HYWELT7GH4LX type=del 2023-12-03T11:48:43.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.257+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=W7IU42CAXKZF type=del size=1260698 2023-12-03T11:48:43.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.258+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~PZRMPBOQ0SN1.0 size=179332 2023-12-03T11:48:43.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.260+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=PZRMPBOQ0SN1 tag=1GES16JI79GB type=write size=1227908 2023-12-03T11:48:43.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.262+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~5W5SCK49XU5W.0 size=1048576 2023-12-03T11:48:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.263+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~5W5SCK49XU5W.1 size=802249 2023-12-03T11:48:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.263+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=5W5SCK49XU5W tag=2AWIU0UXFTF8 type=write size=2899401 2023-12-03T11:48:43.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.265+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=1O0MTXAYLSV1 type=del 2023-12-03T11:48:43.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.266+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=X2XWPDWUUJS4 type=del size=1198335 2023-12-03T11:48:43.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.267+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=JGZJEP62C6CF type=del 2023-12-03T11:48:43.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.270+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.0 size=1048576 2023-12-03T11:48:43.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.271+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.1 size=1048576 2023-12-03T11:48:43.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.271+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.2 size=807635 2023-12-03T11:48:43.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.271+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=KK9PW99VMZH2 tag=Z3VV3VFCDJH6 type=write size=3953363 2023-12-03T11:48:43.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.273+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.0 size=1048576 2023-12-03T11:48:43.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.274+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.1 size=1048576 2023-12-03T11:48:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.275+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~DUF4C7MMML9A.2 size=306894 2023-12-03T11:48:43.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.277+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=DUF4C7MMML9A tag=KZEMSL88ZP2G type=write size=3452622 2023-12-03T11:48:43.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.278+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=1K1MNL9J6EJP type=write size=197027 2023-12-03T11:48:43.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.280+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=7RCDUC3YYY00 type=write size=480622 2023-12-03T11:48:43.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.283+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=VB0XP09S1ILN type=del size=3973564 2023-12-03T11:48:43.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.286+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~5BZE6TLUB15Q.0 size=1048576 2023-12-03T11:48:43.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.287+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~5BZE6TLUB15Q.1 size=391205 2023-12-03T11:48:43.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.289+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=5BZE6TLUB15Q tag=IVF13LMNY0ME type=write size=2488357 2023-12-03T11:48:43.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.290+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=GG13IVPSM3C9 type=del size=2759267 2023-12-03T11:48:43.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.291+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=LD16JO0R55KK type=del size=3981222 2023-12-03T11:48:43.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.294+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.0 size=1048576 2023-12-03T11:48:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.295+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.1 size=1048576 2023-12-03T11:48:43.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.296+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.2 size=320133 2023-12-03T11:48:43.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.296+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=WPOKEMQDDAOM tag=QAOB1XX1EDQK type=write size=3465861 2023-12-03T11:48:43.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.297+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=Q45LT80SIM53 type=del 2023-12-03T11:48:43.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.298+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=PM6YFBO3PMQB type=del size=1269071 2023-12-03T11:48:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.300+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.0 size=1048576 2023-12-03T11:48:43.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.301+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.1 size=1048576 2023-12-03T11:48:43.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.302+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~7ZEKHUW9HRP1.2 size=165707 2023-12-03T11:48:43.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.303+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=7ZEKHUW9HRP1 tag=9C1S8WIDVO5I type=write size=3311435 2023-12-03T11:48:43.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.306+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~P50YZ8VF4GQM.0 size=274263 2023-12-03T11:48:43.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.306+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=P50YZ8VF4GQM tag=91Q2CV6W023M type=write size=1322839 2023-12-03T11:48:43.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.307+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~I46YWX5UFIBI.0 size=572844 2023-12-03T11:48:43.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.309+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=I46YWX5UFIBI tag=IOWQ24NOBW9I type=write size=1621420 2023-12-03T11:48:43.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.312+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~E73A9IE14H3C.0 size=1048576 2023-12-03T11:48:43.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.313+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~E73A9IE14H3C.1 size=656985 2023-12-03T11:48:43.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.313+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=E73A9IE14H3C tag=NEK5FC635G4P type=write size=2754137 2023-12-03T11:48:43.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.314+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=TMLEJ7DAL5T3 type=del 2023-12-03T11:48:43.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.315+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=O2RCKD60U8YQ type=del size=3947199 2023-12-03T11:48:43.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.316+0000 7f24574f5780 1 < completed write operation key=3EOEF9UMATIE tag=VTAB0X2L5T1R type=write size=91677 2023-12-03T11:48:43.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.318+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=0B2LCOECORCF type=del size=4027295 2023-12-03T11:48:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.320+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=XKGU53NN1OPA type=del 2023-12-03T11:48:43.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.322+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~LEWV5LHGQAWK.0 size=425913 2023-12-03T11:48:43.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.322+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=LEWV5LHGQAWK tag=9CCJL67KX3SI type=write size=1474489 2023-12-03T11:48:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.323+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~8P5QPK37LNDP.0 size=1048576 2023-12-03T11:48:43.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.324+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~8P5QPK37LNDP.1 size=651546 2023-12-03T11:48:43.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.326+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=8P5QPK37LNDP tag=5QJNHGUL0FEH type=write size=2748698 2023-12-03T11:48:43.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.327+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=PGLWD6KU34QT type=write size=831224 2023-12-03T11:48:43.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.329+0000 7f24574f5780 1 < suggested remove operation key=XRHVYVADRWET tag=L3YP6ZVC9QW1 type=del 2023-12-03T11:48:43.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.331+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=HOOD4E2DTZP7 type=write size=898706 2023-12-03T11:48:43.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.333+0000 7f24574f5780 1 < suggested update operation key=SC8UOVLQQ0VD tag=W7IU42CAXKZF type=del size=1260698 2023-12-03T11:48:43.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.334+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=ZEZE5IWI5RM2 type=del size=2564634 2023-12-03T11:48:43.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.335+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~5W5SCK49XU5W.0 size=1048576 2023-12-03T11:48:43.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.336+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~5W5SCK49XU5W.1 size=802249 2023-12-03T11:48:43.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.338+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=5W5SCK49XU5W tag=2AWIU0UXFTF8 type=write size=2899401 2023-12-03T11:48:43.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.339+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=ANPT0852CFBB type=del size=691906 2023-12-03T11:48:43.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.339+0000 7f24574f5780 -1 < no bi entry to suggest for operation key=XRHVYVADRWET tag=X2XWPDWUUJS4 type=del size=1198335: (2) No such file or directory 2023-12-03T11:48:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.341+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=MZ30WC7G27P3 type=del size=577449 2023-12-03T11:48:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.342+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.0 size=1048576 2023-12-03T11:48:43.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.343+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.1 size=1048576 2023-12-03T11:48:43.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.344+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~KK9PW99VMZH2.2 size=807635 2023-12-03T11:48:43.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.346+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=KK9PW99VMZH2 tag=Z3VV3VFCDJH6 type=write size=3953363 2023-12-03T11:48:43.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.347+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=YEU9UZNPSC7H type=del size=1335411 2023-12-03T11:48:43.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.348+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=1K1MNL9J6EJP type=write size=197027 2023-12-03T11:48:43.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.349+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=9TZBB52TV26E type=del size=2808930 2023-12-03T11:48:43.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.350+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=VB0XP09S1ILN type=del 2023-12-03T11:48:43.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.351+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=2W4PHYCLFTIB type=del size=3979050 2023-12-03T11:48:43.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.353+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=GG13IVPSM3C9 type=del 2023-12-03T11:48:43.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.355+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.0 size=1048576 2023-12-03T11:48:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.356+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.1 size=1048576 2023-12-03T11:48:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.356+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.2 size=460725 2023-12-03T11:48:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.356+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=CIW6MEM17A4N tag=BLRBCCR3JJYA type=write size=3606453 2023-12-03T11:48:43.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.358+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.0 size=1048576 2023-12-03T11:48:43.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.359+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.1 size=1048576 2023-12-03T11:48:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.360+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~WPOKEMQDDAOM.2 size=320133 2023-12-03T11:48:43.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.362+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=WPOKEMQDDAOM tag=QAOB1XX1EDQK type=write size=3465861 2023-12-03T11:48:43.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.363+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=9BE2K9YCZ756 type=del size=1011473 2023-12-03T11:48:43.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.365+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=PM6YFBO3PMQB type=del size=1269071 2023-12-03T11:48:43.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.367+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.0 size=1048576 2023-12-03T11:48:43.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.368+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.1 size=1048576 2023-12-03T11:48:43.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.369+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.2 size=960761 2023-12-03T11:48:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.369+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=I17UIUI8XNAG tag=W1TTTS64M8BI type=write size=4106489 2023-12-03T11:48:43.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.371+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~P50YZ8VF4GQM.0 size=274263 2023-12-03T11:48:43.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.373+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=P50YZ8VF4GQM tag=91Q2CV6W023M type=write size=1322839 2023-12-03T11:48:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.375+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~3L0OTNCCNLNS.0 size=194948 2023-12-03T11:48:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.375+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=3L0OTNCCNLNS tag=XO0VLEIC0U83 type=write size=1243524 2023-12-03T11:48:43.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.376+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~E73A9IE14H3C.0 size=1048576 2023-12-03T11:48:43.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.377+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~E73A9IE14H3C.1 size=656985 2023-12-03T11:48:43.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.379+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=E73A9IE14H3C tag=NEK5FC635G4P type=write size=2754137 2023-12-03T11:48:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.380+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=T9WHJ7XZ2I4L type=del size=1679039 2023-12-03T11:48:43.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.381+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=O2RCKD60U8YQ type=del 2023-12-03T11:48:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.382+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=6KG226YAJ6HR type=del size=445022 2023-12-03T11:48:43.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.384+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=0B2LCOECORCF type=del 2023-12-03T11:48:43.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.385+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=EGJEF6WSP9BD type=del size=1382160 2023-12-03T11:48:43.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.386+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~LEWV5LHGQAWK.0 size=425913 2023-12-03T11:48:43.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.388+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=LEWV5LHGQAWK tag=9CCJL67KX3SI type=write size=1474489 2023-12-03T11:48:43.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.389+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=5TRBICB8D2G6 type=del size=2525778 2023-12-03T11:48:43.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.391+0000 7f24574f5780 1 < suggested remove operation key=863EOZDYRWH3 tag=PGLWD6KU34QT type=write 2023-12-03T11:48:43.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.393+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=HDSOF0VU5HQI type=write size=1026705 2023-12-03T11:48:43.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.395+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=HOOD4E2DTZP7 type=write size=898706 2023-12-03T11:48:43.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.396+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=IPQTFXYSXMZ5 type=del size=1236309 2023-12-03T11:48:43.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.397+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=ZEZE5IWI5RM2 type=del 2023-12-03T11:48:43.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.400+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~NO2EI58J8ZPY.0 size=654076 2023-12-03T11:48:43.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.400+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=NO2EI58J8ZPY tag=P4DHHPQ3EL6Q type=write size=1702652 2023-12-03T11:48:43.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.401+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=ANPT0852CFBB type=del 2023-12-03T11:48:43.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.402+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=K0HVGENJQZ03 type=del size=3684183 2023-12-03T11:48:43.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.403+0000 7f24574f5780 1 < canceled operation key=XZFHFS76WNR4 tag=MZ30WC7G27P3 type=del size=577449 2023-12-03T11:48:43.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.406+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~HBOAS29WNQZG.0 size=588109 2023-12-03T11:48:43.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.406+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=HBOAS29WNQZG tag=Q1IZI34XJKZ2 type=write size=1636685 2023-12-03T11:48:43.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.407+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=YEU9UZNPSC7H type=del 2023-12-03T11:48:43.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.409+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~UG6RXZR22G0N.0 size=532574 2023-12-03T11:48:43.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.409+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=UG6RXZR22G0N tag=VPXFY7C01XL1 type=write size=1581150 2023-12-03T11:48:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.410+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=9TZBB52TV26E type=del 2023-12-03T11:48:43.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.412+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.0 size=1048576 2023-12-03T11:48:43.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.413+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.1 size=1048576 2023-12-03T11:48:43.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.414+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.2 size=637511 2023-12-03T11:48:43.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.414+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=9SURA7YO7O3A tag=3D3CMC4Q8W5J type=write size=3783239 2023-12-03T11:48:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.417+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=2W4PHYCLFTIB type=del 2023-12-03T11:48:43.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.419+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~LF72R85NRJ7U.0 size=1048576 2023-12-03T11:48:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.420+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~LF72R85NRJ7U.1 size=583636 2023-12-03T11:48:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.420+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=LF72R85NRJ7U tag=FZ1NQME0FHGV type=write size=2680788 2023-12-03T11:48:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.421+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.0 size=1048576 2023-12-03T11:48:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.422+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.1 size=1048576 2023-12-03T11:48:43.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.423+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~CIW6MEM17A4N.2 size=460725 2023-12-03T11:48:43.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.425+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=CIW6MEM17A4N tag=BLRBCCR3JJYA type=write size=3606453 2023-12-03T11:48:43.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.426+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=FZE694DHLHMB type=del size=2773180 2023-12-03T11:48:43.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.427+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=9BE2K9YCZ756 type=del 2023-12-03T11:48:43.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.429+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~IOIFNKJTE17P.0 size=1048576 2023-12-03T11:48:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.430+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~IOIFNKJTE17P.1 size=832526 2023-12-03T11:48:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.430+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=IOIFNKJTE17P tag=GKQOK29HXIPU type=write size=2929678 2023-12-03T11:48:43.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.431+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.0 size=1048576 2023-12-03T11:48:43.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.432+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.1 size=1048576 2023-12-03T11:48:43.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.433+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~I17UIUI8XNAG.2 size=960761 2023-12-03T11:48:43.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.435+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=I17UIUI8XNAG tag=W1TTTS64M8BI type=write size=4106489 2023-12-03T11:48:43.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.436+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=NG3J4IXHWU2Y type=del size=3130690 2023-12-03T11:48:43.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.437+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~3L0OTNCCNLNS.0 size=194948 2023-12-03T11:48:43.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.439+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=3L0OTNCCNLNS tag=XO0VLEIC0U83 type=write size=1243524 2023-12-03T11:48:43.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.441+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.0 size=1048576 2023-12-03T11:48:43.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.442+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.1 size=1048576 2023-12-03T11:48:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.443+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.2 size=97484 2023-12-03T11:48:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.443+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=4IRZCMVSP8W8 tag=J8XCHIEGB0ZH type=write size=3243212 2023-12-03T11:48:43.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.444+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=T9WHJ7XZ2I4L type=del 2023-12-03T11:48:43.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.445+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=IPDGWOY7ESI3 type=del size=593263 2023-12-03T11:48:43.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.446+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=6KG226YAJ6HR type=del 2023-12-03T11:48:43.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.448+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=NYCE5GC5HXG9 type=del size=592368 2023-12-03T11:48:43.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.449+0000 7f24574f5780 1 < canceled operation key=1QR0TE9DXU5Z tag=EGJEF6WSP9BD type=del size=1382160 2023-12-03T11:48:43.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.450+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=71S4IVMTT2FD type=del size=3826046 2023-12-03T11:48:43.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.451+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=5TRBICB8D2G6 type=del 2023-12-03T11:48:43.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.452+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=YNP20T3GPSCR type=del size=3856984 2023-12-03T11:48:43.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.454+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=HDSOF0VU5HQI type=write size=1026705 2023-12-03T11:48:43.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.457+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~4SAWPNX5L7AE.0 size=745860 2023-12-03T11:48:43.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.457+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=4SAWPNX5L7AE tag=AA3EZT5I3YUG type=write size=1794436 2023-12-03T11:48:43.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.458+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=IPQTFXYSXMZ5 type=del 2023-12-03T11:48:43.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.460+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~4FP9B7QMVR6F.0 size=565777 2023-12-03T11:48:43.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.460+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=4FP9B7QMVR6F tag=2VE4O71L6HU6 type=write size=1614353 2023-12-03T11:48:43.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.461+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~NO2EI58J8ZPY.0 size=654076 2023-12-03T11:48:43.464 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.463+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=NO2EI58J8ZPY tag=P4DHHPQ3EL6Q type=write size=1702652 2023-12-03T11:48:43.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.464+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=GTCFQ4CNDWWL type=del size=2034972 2023-12-03T11:48:43.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.465+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=K0HVGENJQZ03 type=del 2023-12-03T11:48:43.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.467+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.0 size=1048576 2023-12-03T11:48:43.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.468+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.1 size=1048576 2023-12-03T11:48:43.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.469+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.2 size=899964 2023-12-03T11:48:43.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.469+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=RQYWIWXAYC01 tag=3R2E3EDFDW2N type=write size=4045692 2023-12-03T11:48:43.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.470+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~HBOAS29WNQZG.0 size=588109 2023-12-03T11:48:43.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.472+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=HBOAS29WNQZG tag=Q1IZI34XJKZ2 type=write size=1636685 2023-12-03T11:48:43.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.474+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=F1XXTVMM7BMN type=del size=3385956 2023-12-03T11:48:43.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.475+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~UG6RXZR22G0N.0 size=532574 2023-12-03T11:48:43.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.477+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=UG6RXZR22G0N tag=VPXFY7C01XL1 type=write size=1581150 2023-12-03T11:48:43.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.478+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=NK098V1JVKCR type=del size=3086882 2023-12-03T11:48:43.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.479+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.0 size=1048576 2023-12-03T11:48:43.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.480+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.1 size=1048576 2023-12-03T11:48:43.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.481+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~9SURA7YO7O3A.2 size=637511 2023-12-03T11:48:43.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.483+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=9SURA7YO7O3A tag=3D3CMC4Q8W5J type=write size=3783239 2023-12-03T11:48:43.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.484+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=GYKJ8FBRCKTZ type=del size=2672547 2023-12-03T11:48:43.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.485+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~LF72R85NRJ7U.0 size=1048576 2023-12-03T11:48:43.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.486+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~LF72R85NRJ7U.1 size=583636 2023-12-03T11:48:43.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.488+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=LF72R85NRJ7U tag=FZ1NQME0FHGV type=write size=2680788 2023-12-03T11:48:43.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.489+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=GSOYPKOI1BYW type=del size=1190218 2023-12-03T11:48:43.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.491+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=FZE694DHLHMB type=del size=2773180 2023-12-03T11:48:43.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.492+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=PO3XT6V766T3 type=del size=2906308 2023-12-03T11:48:43.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.493+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~IOIFNKJTE17P.0 size=1048576 2023-12-03T11:48:43.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.494+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~IOIFNKJTE17P.1 size=832526 2023-12-03T11:48:43.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.496+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=IOIFNKJTE17P tag=GKQOK29HXIPU type=write size=2929678 2023-12-03T11:48:43.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.498+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.0 size=1048576 2023-12-03T11:48:43.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.499+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.1 size=1048576 2023-12-03T11:48:43.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.500+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.2 size=511424 2023-12-03T11:48:43.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.500+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=GTO2KM2ODTE3 tag=AG9XCYKG3381 type=write size=3657152 2023-12-03T11:48:43.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.501+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=NG3J4IXHWU2Y type=del 2023-12-03T11:48:43.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.504+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~689CNBGE9LUW.0 size=1048576 2023-12-03T11:48:43.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.506+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~689CNBGE9LUW.1 size=328741 2023-12-03T11:48:43.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.506+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=689CNBGE9LUW tag=1H8OWP4IEK4U type=write size=2425893 2023-12-03T11:48:43.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.508+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.0 size=1048576 2023-12-03T11:48:43.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.509+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.1 size=1048576 2023-12-03T11:48:43.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.510+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~4IRZCMVSP8W8.2 size=97484 2023-12-03T11:48:43.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.512+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=4IRZCMVSP8W8 tag=J8XCHIEGB0ZH type=write size=3243212 2023-12-03T11:48:43.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.513+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=G7AWIKPDN5GM type=del size=3838502 2023-12-03T11:48:43.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.516+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=IPDGWOY7ESI3 type=del 2023-12-03T11:48:43.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.518+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.0 size=1048576 2023-12-03T11:48:43.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.519+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.1 size=1048576 2023-12-03T11:48:43.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.520+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.2 size=964311 2023-12-03T11:48:43.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.520+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=ZXSH25BLRSA5 tag=0YLIS6EMOMME type=write size=4110039 2023-12-03T11:48:43.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.522+0000 7f24574f5780 1 < suggested update operation key=863EOZDYRWH3 tag=NYCE5GC5HXG9 type=del size=592368 2023-12-03T11:48:43.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.524+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.0 size=1048576 2023-12-03T11:48:43.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.525+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.1 size=1048576 2023-12-03T11:48:43.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.526+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.2 size=34985 2023-12-03T11:48:43.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.526+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=WR5XY8SJWE1Z tag=LQG3P1DO4R65 type=write size=3180713 2023-12-03T11:48:43.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.527+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=71S4IVMTT2FD type=del 2023-12-03T11:48:43.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.528+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=0O8PR4AZQPMX type=del size=3753413 2023-12-03T11:48:43.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.529+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=YNP20T3GPSCR type=del 2023-12-03T11:48:43.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.531+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~UHZ3VDR3OI7M.0 size=60209 2023-12-03T11:48:43.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.531+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=UHZ3VDR3OI7M tag=3I11VR2EAG0H type=write size=1108785 2023-12-03T11:48:43.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.532+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~4SAWPNX5L7AE.0 size=745860 2023-12-03T11:48:43.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.534+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=4SAWPNX5L7AE tag=AA3EZT5I3YUG type=write size=1794436 2023-12-03T11:48:43.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.535+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=2XNBBLEFVM17 type=write size=704637 2023-12-03T11:48:43.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.535+0000 7f24574f5780 1 < canceled part key=_multipart_SC8UOVLQQ0VD.2~4FP9B7QMVR6F.0 size=565777 2023-12-03T11:48:43.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.537+0000 7f24574f5780 1 < canceled multipart upload key=SC8UOVLQQ0VD upload=4FP9B7QMVR6F tag=2VE4O71L6HU6 type=write size=1614353 2023-12-03T11:48:43.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.539+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~DZ60E100BUKD.0 size=81248 2023-12-03T11:48:43.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.539+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=DZ60E100BUKD tag=PGNTA493BOP4 type=write size=1129824 2023-12-03T11:48:43.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.540+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=GTCFQ4CNDWWL type=del 2023-12-03T11:48:43.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.542+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=3YGFSAJCANKO type=write size=402551 2023-12-03T11:48:43.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.543+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.0 size=1048576 2023-12-03T11:48:43.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.544+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.1 size=1048576 2023-12-03T11:48:43.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.544+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~RQYWIWXAYC01.2 size=899964 2023-12-03T11:48:43.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.546+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=RQYWIWXAYC01 tag=3R2E3EDFDW2N type=write size=4045692 2023-12-03T11:48:43.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.547+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=H245445PTHF6 type=del size=1406399 2023-12-03T11:48:43.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.549+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=F1XXTVMM7BMN type=del 2023-12-03T11:48:43.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.550+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=GAHX08MMDKU6 type=del size=950853 2023-12-03T11:48:43.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.551+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=NK098V1JVKCR type=del 2023-12-03T11:48:43.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.552+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=OPW9S01WZFMB type=write size=590356 2023-12-03T11:48:43.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.554+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=GYKJ8FBRCKTZ type=del 2023-12-03T11:48:43.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.555+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=DRGPZT4WYOW7 type=del size=296413 2023-12-03T11:48:43.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.556+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=GSOYPKOI1BYW type=del 2023-12-03T11:48:43.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.558+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5UQ1HG94PAUN.0 size=1048576 2023-12-03T11:48:43.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.559+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~5UQ1HG94PAUN.1 size=228754 2023-12-03T11:48:43.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.559+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=5UQ1HG94PAUN tag=PK9NYZYZIK01 type=write size=2325906 2023-12-03T11:48:43.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.560+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=PO3XT6V766T3 type=del 2023-12-03T11:48:43.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.562+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=XRY0EJ11ZNQ6 type=del size=340956 2023-12-03T11:48:43.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.563+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.0 size=1048576 2023-12-03T11:48:43.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.564+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.1 size=1048576 2023-12-03T11:48:43.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.565+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~GTO2KM2ODTE3.2 size=511424 2023-12-03T11:48:43.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.567+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=GTO2KM2ODTE3 tag=AG9XCYKG3381 type=write size=3657152 2023-12-03T11:48:43.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.570+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~VUP74WAO6SLF.0 size=1048576 2023-12-03T11:48:43.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.571+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~VUP74WAO6SLF.1 size=257024 2023-12-03T11:48:43.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.571+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=VUP74WAO6SLF tag=XPL9W6B4Y9RQ type=write size=2354176 2023-12-03T11:48:43.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.572+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~689CNBGE9LUW.0 size=1048576 2023-12-03T11:48:43.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.573+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~689CNBGE9LUW.1 size=328741 2023-12-03T11:48:43.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.576+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=689CNBGE9LUW tag=1H8OWP4IEK4U type=write size=2425893 2023-12-03T11:48:43.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.577+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=JQYTN1PTSJK9 type=del size=4185593 2023-12-03T11:48:43.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.578+0000 7f24574f5780 -1 < failed to complete operation key=A4LOQE7VRDN7 tag=G7AWIKPDN5GM type=del size=3838502: (22) Invalid argument 2023-12-03T11:48:43.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.579+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=UCINX809H8TS type=del size=2388007 2023-12-03T11:48:43.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.580+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.0 size=1048576 2023-12-03T11:48:43.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.581+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.1 size=1048576 2023-12-03T11:48:43.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.582+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZXSH25BLRSA5.2 size=964311 2023-12-03T11:48:43.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.584+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=ZXSH25BLRSA5 tag=0YLIS6EMOMME type=write size=4110039 2023-12-03T11:48:43.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.586+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=WOANMNTZX2NR type=del size=2321291 2023-12-03T11:48:43.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.587+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.0 size=1048576 2023-12-03T11:48:43.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.588+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.1 size=1048576 2023-12-03T11:48:43.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.589+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~WR5XY8SJWE1Z.2 size=34985 2023-12-03T11:48:43.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.591+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=WR5XY8SJWE1Z tag=LQG3P1DO4R65 type=write size=3180713 2023-12-03T11:48:43.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.592+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=TCTHGM8X00A0 type=del size=477886 2023-12-03T11:48:43.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.594+0000 7f24574f5780 1 < suggested update operation key=DA6ZXCLKTC1L tag=0O8PR4AZQPMX type=del size=3753413 2023-12-03T11:48:43.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.595+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=0ST6N07MF1Y2 type=del size=2544455 2023-12-03T11:48:43.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.597+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~UHZ3VDR3OI7M.0 size=60209 2023-12-03T11:48:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.599+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=UHZ3VDR3OI7M tag=3I11VR2EAG0H type=write size=1108785 2023-12-03T11:48:43.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.600+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=UZNJKPRJ3SHZ type=del size=2581696 2023-12-03T11:48:43.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.601+0000 7f24574f5780 1 < completed write operation key=XO42YZR60KQE tag=2XNBBLEFVM17 type=write size=704637 2023-12-03T11:48:43.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.602+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=TA1MOHAKW0CI type=del size=1269567 2023-12-03T11:48:43.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.603+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~DZ60E100BUKD.0 size=81248 2023-12-03T11:48:43.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.605+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=DZ60E100BUKD tag=PGNTA493BOP4 type=write size=1129824 2023-12-03T11:48:43.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.606+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=FID8LP98CVRY type=del size=3684677 2023-12-03T11:48:43.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.607+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=3YGFSAJCANKO type=write size=402551 2023-12-03T11:48:43.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.608+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=0X05JXOEG8Y4 type=write size=592277 2023-12-03T11:48:43.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.610+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=H245445PTHF6 type=del 2023-12-03T11:48:43.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.611+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=37KJGR1S45ED type=write size=927218 2023-12-03T11:48:43.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.613+0000 7f24574f5780 1 < suggested update operation key=QVLPCWC4T8N8 tag=GAHX08MMDKU6 type=del size=950853 2023-12-03T11:48:43.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.615+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=7PDSTXQUTAOK type=write size=124317 2023-12-03T11:48:43.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.617+0000 7f24574f5780 1 < suggested update operation key=MY3K9M4QRS3L tag=OPW9S01WZFMB type=write size=590356 2023-12-03T11:48:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.619+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=LOWC46RJA137 type=write size=194466 2023-12-03T11:48:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.620+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=DRGPZT4WYOW7 type=del 2023-12-03T11:48:43.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.622+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.0 size=1048576 2023-12-03T11:48:43.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.623+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.1 size=1048576 2023-12-03T11:48:43.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.624+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.2 size=197159 2023-12-03T11:48:43.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.624+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=RDKNGWC50705 tag=7KDPJ7ZUINFR type=write size=3342887 2023-12-03T11:48:43.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.625+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5UQ1HG94PAUN.0 size=1048576 2023-12-03T11:48:43.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.626+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~5UQ1HG94PAUN.1 size=228754 2023-12-03T11:48:43.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.628+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=5UQ1HG94PAUN tag=PK9NYZYZIK01 type=write size=2325906 2023-12-03T11:48:43.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.630+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.0 size=1048576 2023-12-03T11:48:43.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.631+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.1 size=1048576 2023-12-03T11:48:43.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.632+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.2 size=611695 2023-12-03T11:48:43.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.632+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=M047XBCY6E0P tag=4F3PDLS2Y04K type=write size=3757423 2023-12-03T11:48:43.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.633+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=XRY0EJ11ZNQ6 type=del 2023-12-03T11:48:43.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.634+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=Q8YP94RC8IC4 type=del size=951253 2023-12-03T11:48:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.635+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~VUP74WAO6SLF.0 size=1048576 2023-12-03T11:48:43.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.636+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~VUP74WAO6SLF.1 size=257024 2023-12-03T11:48:43.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.638+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=VUP74WAO6SLF tag=XPL9W6B4Y9RQ type=write size=2354176 2023-12-03T11:48:43.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.640+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=J3USGIRS2LTM type=del size=1634051 2023-12-03T11:48:43.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.641+0000 7f24574f5780 1 < canceled operation key=35U7YCDYMX46 tag=JQYTN1PTSJK9 type=del size=4185593 2023-12-03T11:48:43.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.642+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=4B7NNNY90SE9 type=del size=3086791 2023-12-03T11:48:43.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.643+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=UCINX809H8TS type=del 2023-12-03T11:48:43.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.645+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=U95XV8JXBLX5 type=del size=3328108 2023-12-03T11:48:43.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.646+0000 7f24574f5780 1 < canceled operation key=S7Q5TCM499H1 tag=WOANMNTZX2NR type=del size=2321291 2023-12-03T11:48:43.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.648+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~XT6GRS7EYF3V.0 size=306883 2023-12-03T11:48:43.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.648+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=XT6GRS7EYF3V tag=2WD57OWRC6FS type=write size=1355459 2023-12-03T11:48:43.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.649+0000 7f24574f5780 -1 < failed to complete operation key=MY3K9M4QRS3L tag=TCTHGM8X00A0 type=del size=477886: (22) Invalid argument 2023-12-03T11:48:43.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.651+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~U5RM2JGXWBQB.0 size=1048576 2023-12-03T11:48:43.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.652+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~U5RM2JGXWBQB.1 size=293793 2023-12-03T11:48:43.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.652+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=U5RM2JGXWBQB tag=UTD69FF9G4MF type=write size=2390945 2023-12-03T11:48:43.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.653+0000 7f24574f5780 1 < canceled operation key=PS1OYWI44L80 tag=0ST6N07MF1Y2 type=del size=2544455 2023-12-03T11:48:43.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.655+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~29M85E08L9NN.0 size=1048576 2023-12-03T11:48:43.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.656+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~29M85E08L9NN.1 size=13307 2023-12-03T11:48:43.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.656+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=29M85E08L9NN tag=NZCEEGW1HK7C type=write size=2110459 2023-12-03T11:48:43.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.657+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=UZNJKPRJ3SHZ type=del 2023-12-03T11:48:43.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.659+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~RKX85ZVLG3Z1.0 size=311218 2023-12-03T11:48:43.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.659+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=RKX85ZVLG3Z1 tag=L8WETC0UOO5A type=write size=1359794 2023-12-03T11:48:43.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.662+0000 7f24574f5780 1 < suggested update operation key=MH1SRAATMNXB tag=TA1MOHAKW0CI type=del size=1269567 2023-12-03T11:48:43.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.664+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~G35EGBN8BDE7.0 size=604929 2023-12-03T11:48:43.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.664+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=G35EGBN8BDE7 tag=OUX0UXUFL3LX type=write size=1653505 2023-12-03T11:48:43.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.665+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=FID8LP98CVRY type=del 2023-12-03T11:48:43.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.666+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=UTH4DP7EPJCI type=del size=396105 2023-12-03T11:48:43.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.668+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=0X05JXOEG8Y4 type=write size=592277 2023-12-03T11:48:43.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.671+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~YRY1YX24SKWM.0 size=1048576 2023-12-03T11:48:43.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.671+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~YRY1YX24SKWM.1 size=290579 2023-12-03T11:48:43.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.671+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=YRY1YX24SKWM tag=ZV1QLVBY46U4 type=write size=2387731 2023-12-03T11:48:43.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.674+0000 7f24574f5780 1 < suggested update operation key=SC8UOVLQQ0VD tag=37KJGR1S45ED type=write size=927218 2023-12-03T11:48:43.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.676+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~W8X05KKQPO8X.0 size=426982 2023-12-03T11:48:43.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.676+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=W8X05KKQPO8X tag=V9VO9YLTE11W type=write size=1475558 2023-12-03T11:48:43.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.678+0000 7f24574f5780 1 < suggested update operation key=9LKFY1U0JRBQ tag=7PDSTXQUTAOK type=write size=124317 2023-12-03T11:48:43.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.680+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~MGOG9WS34I6C.0 size=919318 2023-12-03T11:48:43.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.680+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=MGOG9WS34I6C tag=41DQ8L211WME type=write size=1967894 2023-12-03T11:48:43.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.682+0000 7f24574f5780 1 < suggested update operation key=WZMWCSAZX8RB tag=LOWC46RJA137 type=write size=194466 2023-12-03T11:48:43.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.683+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=FET9JX291MRT type=del size=1296526 2023-12-03T11:48:43.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.684+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.0 size=1048576 2023-12-03T11:48:43.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.685+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.1 size=1048576 2023-12-03T11:48:43.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.686+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~RDKNGWC50705.2 size=197159 2023-12-03T11:48:43.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.688+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=RDKNGWC50705 tag=7KDPJ7ZUINFR type=write size=3342887 2023-12-03T11:48:43.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.689+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=5B0N41FIGE8I type=write size=1017162 2023-12-03T11:48:43.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.690+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.0 size=1048576 2023-12-03T11:48:43.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.691+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.1 size=1048576 2023-12-03T11:48:43.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.692+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~M047XBCY6E0P.2 size=611695 2023-12-03T11:48:43.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.694+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=M047XBCY6E0P tag=4F3PDLS2Y04K type=write size=3757423 2023-12-03T11:48:43.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.696+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~9AQMQHHSK5ZL.0 size=1048576 2023-12-03T11:48:43.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.697+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~9AQMQHHSK5ZL.1 size=421334 2023-12-03T11:48:43.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.697+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=9AQMQHHSK5ZL tag=46T9TBZZ92RJ type=write size=2518486 2023-12-03T11:48:43.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.700+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=Q8YP94RC8IC4 type=del size=951253 2023-12-03T11:48:43.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.701+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=1K570UUV9W7D type=del size=4087116 2023-12-03T11:48:43.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.702+0000 7f24574f5780 -1 < failed to complete operation key=MH1SRAATMNXB tag=J3USGIRS2LTM type=del size=1634051: (22) Invalid argument 2023-12-03T11:48:43.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.705+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.0 size=1048576 2023-12-03T11:48:43.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.706+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.1 size=1048576 2023-12-03T11:48:43.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.706+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.2 size=582547 2023-12-03T11:48:43.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.706+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=BQEJZC1668JK tag=4UJ8ZE5NFO3F type=write size=3728275 2023-12-03T11:48:43.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.708+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=4B7NNNY90SE9 type=del 2023-12-03T11:48:43.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.709+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=JR16495CCZTI type=del size=542976 2023-12-03T11:48:43.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.710+0000 7f24574f5780 -1 < failed to complete operation key=3EOEF9UMATIE tag=U95XV8JXBLX5 type=del size=3328108: (22) Invalid argument 2023-12-03T11:48:43.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.711+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=F31W7FUKWOTS type=del size=2356208 2023-12-03T11:48:43.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.713+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~XT6GRS7EYF3V.0 size=306883 2023-12-03T11:48:43.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.715+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=XT6GRS7EYF3V tag=2WD57OWRC6FS type=write size=1355459 2023-12-03T11:48:43.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.716+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=377SU5CV8F1D type=del size=1344000 2023-12-03T11:48:43.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.716+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~U5RM2JGXWBQB.0 size=1048576 2023-12-03T11:48:43.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.717+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~U5RM2JGXWBQB.1 size=293793 2023-12-03T11:48:43.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.719+0000 7f24574f5780 1 < canceled multipart upload key=RABUR02D2ZMD upload=U5RM2JGXWBQB tag=UTD69FF9G4MF type=write size=2390945 2023-12-03T11:48:43.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.722+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~5XBYTP8852MC.0 size=563816 2023-12-03T11:48:43.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.722+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=5XBYTP8852MC tag=0UMEXAUVLKLN type=write size=1612392 2023-12-03T11:48:43.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.723+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~29M85E08L9NN.0 size=1048576 2023-12-03T11:48:43.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.725+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~29M85E08L9NN.1 size=13307 2023-12-03T11:48:43.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.727+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=29M85E08L9NN tag=NZCEEGW1HK7C type=write size=2110459 2023-12-03T11:48:43.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.729+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.0 size=1048576 2023-12-03T11:48:43.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.730+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.1 size=1048576 2023-12-03T11:48:43.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.731+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.2 size=135421 2023-12-03T11:48:43.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.731+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=DEWR6CB18MC3 tag=BRTD2RKNPAPQ type=write size=3281149 2023-12-03T11:48:43.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.732+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~RKX85ZVLG3Z1.0 size=311218 2023-12-03T11:48:43.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.734+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=RKX85ZVLG3Z1 tag=L8WETC0UOO5A type=write size=1359794 2023-12-03T11:48:43.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.735+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=MXYRKCPWT344 type=del size=3870082 2023-12-03T11:48:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.736+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~G35EGBN8BDE7.0 size=604929 2023-12-03T11:48:43.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.738+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=G35EGBN8BDE7 tag=OUX0UXUFL3LX type=write size=1653505 2023-12-03T11:48:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.739+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=S3OLIUMS1KK0 type=del size=1010617 2023-12-03T11:48:43.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.741+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=UTH4DP7EPJCI type=del 2023-12-03T11:48:43.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.742+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=04PB4Z1Z1CD0 type=del size=3837941 2023-12-03T11:48:43.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.743+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~YRY1YX24SKWM.0 size=1048576 2023-12-03T11:48:43.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.746+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~YRY1YX24SKWM.1 size=290579 2023-12-03T11:48:43.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.748+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=YRY1YX24SKWM tag=ZV1QLVBY46U4 type=write size=2387731 2023-12-03T11:48:43.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.749+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=LK0IGOKCPWZ3 type=write size=397584 2023-12-03T11:48:43.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.749+0000 7f24574f5780 1 < canceled part key=_multipart_A4LOQE7VRDN7.2~W8X05KKQPO8X.0 size=426982 2023-12-03T11:48:43.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.751+0000 7f24574f5780 1 < canceled multipart upload key=A4LOQE7VRDN7 upload=W8X05KKQPO8X tag=V9VO9YLTE11W type=write size=1475558 2023-12-03T11:48:43.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.752+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=3G08F12F75YX type=write size=540976 2023-12-03T11:48:43.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.753+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~MGOG9WS34I6C.0 size=919318 2023-12-03T11:48:43.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.755+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=MGOG9WS34I6C tag=41DQ8L211WME type=write size=1967894 2023-12-03T11:48:43.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.757+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~WXB3RWULAKXR.0 size=712661 2023-12-03T11:48:43.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.757+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=WXB3RWULAKXR tag=U0FX8IB7DGJP type=write size=1761237 2023-12-03T11:48:43.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.758+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=FET9JX291MRT type=del 2023-12-03T11:48:43.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.760+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=1Z4VPCCZ5UHQ type=del size=4091116 2023-12-03T11:48:43.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.761+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=5B0N41FIGE8I type=write size=1017162 2023-12-03T11:48:43.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.763+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~8SI15TNL47ZS.0 size=452185 2023-12-03T11:48:43.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.763+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=8SI15TNL47ZS tag=ESY65G89SOXW type=write size=1500761 2023-12-03T11:48:43.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.764+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~9AQMQHHSK5ZL.0 size=1048576 2023-12-03T11:48:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.765+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~9AQMQHHSK5ZL.1 size=421334 2023-12-03T11:48:43.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.767+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=9AQMQHHSK5ZL tag=46T9TBZZ92RJ type=write size=2518486 2023-12-03T11:48:43.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.769+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~SPD0IQO4X42V.0 size=1048576 2023-12-03T11:48:43.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.770+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~SPD0IQO4X42V.1 size=941218 2023-12-03T11:48:43.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.770+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=SPD0IQO4X42V tag=IAQD4YJIUZBU type=write size=3038370 2023-12-03T11:48:43.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.772+0000 7f24574f5780 1 < suggested remove operation key=BV8C0VSJ1GTW tag=1K570UUV9W7D type=del 2023-12-03T11:48:43.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.773+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=N6J9ZI782U0U type=del size=3910120 2023-12-03T11:48:43.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.775+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.0 size=1048576 2023-12-03T11:48:43.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.775+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.1 size=1048576 2023-12-03T11:48:43.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.776+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~BQEJZC1668JK.2 size=582547 2023-12-03T11:48:43.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.779+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=BQEJZC1668JK tag=4UJ8ZE5NFO3F type=write size=3728275 2023-12-03T11:48:43.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.780+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=SY8M1JXT16TZ type=del size=319817 2023-12-03T11:48:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.781+0000 7f24574f5780 -1 < failed to complete operation key=BV8C0VSJ1GTW tag=JR16495CCZTI type=del size=542976: (22) Invalid argument 2023-12-03T11:48:43.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.782+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=N2G513G793O5 type=del size=2690984 2023-12-03T11:48:43.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.783+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=F31W7FUKWOTS type=del 2023-12-03T11:48:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.784+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=4NO93KCBI041 type=del size=373896 2023-12-03T11:48:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.786+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=377SU5CV8F1D type=del 2023-12-03T11:48:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.788+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.0 size=1048576 2023-12-03T11:48:43.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.789+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.1 size=1048576 2023-12-03T11:48:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.789+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.2 size=535244 2023-12-03T11:48:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.789+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=3QRTPHTLN36T tag=0KNNUE14WHG2 type=write size=3680972 2023-12-03T11:48:43.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.791+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~5XBYTP8852MC.0 size=563816 2023-12-03T11:48:43.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.793+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=5XBYTP8852MC tag=0UMEXAUVLKLN type=write size=1612392 2023-12-03T11:48:43.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.794+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=O753C2SXFUAY type=del size=432022 2023-12-03T11:48:43.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.795+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.0 size=1048576 2023-12-03T11:48:43.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.796+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.1 size=1048576 2023-12-03T11:48:43.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.796+0000 7f24574f5780 1 < canceled part key=_multipart_PS1OYWI44L80.2~DEWR6CB18MC3.2 size=135421 2023-12-03T11:48:43.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.798+0000 7f24574f5780 1 < canceled multipart upload key=PS1OYWI44L80 upload=DEWR6CB18MC3 tag=BRTD2RKNPAPQ type=write size=3281149 2023-12-03T11:48:43.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.802+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~T0SK9VRMDT1F.0 size=1048576 2023-12-03T11:48:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.803+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~T0SK9VRMDT1F.1 size=38673 2023-12-03T11:48:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.803+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=T0SK9VRMDT1F tag=VKJU0K89ASC9 type=write size=2135825 2023-12-03T11:48:43.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.804+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=MXYRKCPWT344 type=del 2023-12-03T11:48:43.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.806+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~107802M5F1JD.0 size=1048576 2023-12-03T11:48:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.807+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~107802M5F1JD.1 size=385119 2023-12-03T11:48:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.807+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=107802M5F1JD tag=472NBOPMRK2D type=write size=2482271 2023-12-03T11:48:43.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.810+0000 7f24574f5780 1 < suggested remove operation key=BMKG19L35HLV tag=S3OLIUMS1KK0 type=del 2023-12-03T11:48:43.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.812+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.0 size=1048576 2023-12-03T11:48:43.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.813+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.1 size=1048576 2023-12-03T11:48:43.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.814+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.2 size=805013 2023-12-03T11:48:43.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.814+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=MOHBZAZ1YHGL tag=SRM3YPZVN4GY type=write size=3950741 2023-12-03T11:48:43.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.816+0000 7f24574f5780 -1 < failed to complete operation key=BV8C0VSJ1GTW tag=04PB4Z1Z1CD0 type=del size=3837941: (22) Invalid argument 2023-12-03T11:48:43.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.818+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PLWSSATYBQNP.0 size=1048576 2023-12-03T11:48:43.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.819+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PLWSSATYBQNP.1 size=868251 2023-12-03T11:48:43.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.819+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=PLWSSATYBQNP tag=4SDRJCS39USY type=write size=2965403 2023-12-03T11:48:43.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.820+0000 7f24574f5780 1 < completed write operation key=WD7U31FW3NGZ tag=LK0IGOKCPWZ3 type=write size=397584 2023-12-03T11:48:43.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.822+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~0W7HJKEJHLV3.0 size=460792 2023-12-03T11:48:43.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.822+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=0W7HJKEJHLV3 tag=Q9WGP15W6ZNN type=write size=1509368 2023-12-03T11:48:43.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.823+0000 7f24574f5780 1 < completed write operation key=863EOZDYRWH3 tag=3G08F12F75YX type=write size=540976 2023-12-03T11:48:43.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.825+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=0121BQ6NW7R6 type=del size=1290348 2023-12-03T11:48:43.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.826+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~WXB3RWULAKXR.0 size=712661 2023-12-03T11:48:43.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.828+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=WXB3RWULAKXR tag=U0FX8IB7DGJP type=write size=1761237 2023-12-03T11:48:43.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.830+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~6UHJO4WFXOKM.0 size=232965 2023-12-03T11:48:43.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.830+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=6UHJO4WFXOKM tag=C6MKX20HHR6A type=write size=1281541 2023-12-03T11:48:43.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.832+0000 7f24574f5780 1 < suggested remove operation key=XO42YZR60KQE tag=1Z4VPCCZ5UHQ type=del 2023-12-03T11:48:43.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.834+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~POXZ28586315.0 size=1048576 2023-12-03T11:48:43.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.835+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~POXZ28586315.1 size=348866 2023-12-03T11:48:43.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.835+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=POXZ28586315 tag=5LNP1YE3QLN0 type=write size=2446018 2023-12-03T11:48:43.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.838+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~8SI15TNL47ZS.0 size=452185 2023-12-03T11:48:43.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.840+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=8SI15TNL47ZS tag=ESY65G89SOXW type=write size=1500761 2023-12-03T11:48:43.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.842+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~RWLSSX6R84B7.0 size=545542 2023-12-03T11:48:43.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.842+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=RWLSSX6R84B7 tag=GZDJQPGWM3B2 type=write size=1594118 2023-12-03T11:48:43.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.843+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~SPD0IQO4X42V.0 size=1048576 2023-12-03T11:48:43.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.843+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~SPD0IQO4X42V.1 size=941218 2023-12-03T11:48:43.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.845+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=SPD0IQO4X42V tag=IAQD4YJIUZBU type=write size=3038370 2023-12-03T11:48:43.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.846+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=X79KPERM1DUP type=del size=943036 2023-12-03T11:48:43.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.847+0000 7f24574f5780 -1 < failed to complete operation key=BMKG19L35HLV tag=N6J9ZI782U0U type=del size=3910120: (22) Invalid argument 2023-12-03T11:48:43.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.848+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=39JDSIL2MI91 type=del size=702248 2023-12-03T11:48:43.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.849+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=SY8M1JXT16TZ type=del 2023-12-03T11:48:43.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.851+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.0 size=1048576 2023-12-03T11:48:43.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.852+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.1 size=1048576 2023-12-03T11:48:43.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.853+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.2 size=837262 2023-12-03T11:48:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.853+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=KCGEXGK2THK8 tag=D0M9YCWZJ5JL type=write size=3982990 2023-12-03T11:48:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.854+0000 7f24574f5780 1 < canceled operation key=SC8UOVLQQ0VD tag=N2G513G793O5 type=del size=2690984 2023-12-03T11:48:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.856+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=50NAT97RYU1F type=del size=2960183 2023-12-03T11:48:43.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.857+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=4NO93KCBI041 type=del 2023-12-03T11:48:43.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.859+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.0 size=1048576 2023-12-03T11:48:43.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.860+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.1 size=1048576 2023-12-03T11:48:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.861+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.2 size=513892 2023-12-03T11:48:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.861+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=UG5EEM33C8VS tag=0HKSL3UZA3BO type=write size=3659620 2023-12-03T11:48:43.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.862+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.0 size=1048576 2023-12-03T11:48:43.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.863+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.1 size=1048576 2023-12-03T11:48:43.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.864+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~3QRTPHTLN36T.2 size=535244 2023-12-03T11:48:43.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.866+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=3QRTPHTLN36T tag=0KNNUE14WHG2 type=write size=3680972 2023-12-03T11:48:43.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.868+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~R81557V7I6IW.0 size=1048576 2023-12-03T11:48:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.869+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~R81557V7I6IW.1 size=152604 2023-12-03T11:48:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.869+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=R81557V7I6IW tag=STVSBWXPCMMI type=write size=2249756 2023-12-03T11:48:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.870+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=O753C2SXFUAY type=del 2023-12-03T11:48:43.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.872+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.0 size=1048576 2023-12-03T11:48:43.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.873+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.1 size=1048576 2023-12-03T11:48:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.874+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.2 size=876830 2023-12-03T11:48:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.874+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=JHT55G865J7C tag=Z4CDDQ1WUUQA type=write size=4022558 2023-12-03T11:48:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.875+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~T0SK9VRMDT1F.0 size=1048576 2023-12-03T11:48:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.876+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~T0SK9VRMDT1F.1 size=38673 2023-12-03T11:48:43.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.878+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=T0SK9VRMDT1F tag=VKJU0K89ASC9 type=write size=2135825 2023-12-03T11:48:43.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.880+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~W697I4L1N8WL.0 size=1048576 2023-12-03T11:48:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.881+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~W697I4L1N8WL.1 size=784737 2023-12-03T11:48:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.881+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=W697I4L1N8WL tag=VQ6NB5TM6I80 type=write size=2881889 2023-12-03T11:48:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.882+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~107802M5F1JD.0 size=1048576 2023-12-03T11:48:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.882+0000 7f24574f5780 1 < canceled part key=_multipart_HM9JHOV8W24F.2~107802M5F1JD.1 size=385119 2023-12-03T11:48:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.885+0000 7f24574f5780 1 < canceled multipart upload key=HM9JHOV8W24F upload=107802M5F1JD tag=472NBOPMRK2D type=write size=2482271 2023-12-03T11:48:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.886+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=913EQNWSIWGV type=del size=1459682 2023-12-03T11:48:43.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.888+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.0 size=1048576 2023-12-03T11:48:43.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.889+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.1 size=1048576 2023-12-03T11:48:43.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.889+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~MOHBZAZ1YHGL.2 size=805013 2023-12-03T11:48:43.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.891+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=MOHBZAZ1YHGL tag=SRM3YPZVN4GY type=write size=3950741 2023-12-03T11:48:43.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.893+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=6ASGPSGIUZWH type=del size=640980 2023-12-03T11:48:43.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.894+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PLWSSATYBQNP.0 size=1048576 2023-12-03T11:48:43.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.895+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PLWSSATYBQNP.1 size=868251 2023-12-03T11:48:43.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.897+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=PLWSSATYBQNP tag=4SDRJCS39USY type=write size=2965403 2023-12-03T11:48:43.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.898+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=RSHZSP185KBR type=del size=1128617 2023-12-03T11:48:43.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.899+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~0W7HJKEJHLV3.0 size=460792 2023-12-03T11:48:43.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.902+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=0W7HJKEJHLV3 tag=Q9WGP15W6ZNN type=write size=1509368 2023-12-03T11:48:43.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.904+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.0 size=1048576 2023-12-03T11:48:43.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.905+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.1 size=1048576 2023-12-03T11:48:43.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.906+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.2 size=501582 2023-12-03T11:48:43.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.906+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=W8H3CJLCXYD3 tag=V3FA427JN8J4 type=write size=3647310 2023-12-03T11:48:43.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.907+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=0121BQ6NW7R6 type=del 2023-12-03T11:48:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.909+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.0 size=1048576 2023-12-03T11:48:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.910+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.1 size=1048576 2023-12-03T11:48:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.911+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.2 size=308344 2023-12-03T11:48:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.911+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=EHWT6PKZIGX9 tag=4QRVBERE9JO9 type=write size=3454072 2023-12-03T11:48:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.912+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~6UHJO4WFXOKM.0 size=232965 2023-12-03T11:48:43.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.914+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=6UHJO4WFXOKM tag=C6MKX20HHR6A type=write size=1281541 2023-12-03T11:48:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.916+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~Y3HNYZF3NTDR.0 size=740906 2023-12-03T11:48:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.916+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=Y3HNYZF3NTDR tag=KU2FOGW3QBB4 type=write size=1789482 2023-12-03T11:48:43.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.917+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~POXZ28586315.0 size=1048576 2023-12-03T11:48:43.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.918+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~POXZ28586315.1 size=348866 2023-12-03T11:48:43.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.920+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=POXZ28586315 tag=5LNP1YE3QLN0 type=write size=2446018 2023-12-03T11:48:43.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.921+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=16EP2U9NEW7L type=del size=3856969 2023-12-03T11:48:43.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.922+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~RWLSSX6R84B7.0 size=545542 2023-12-03T11:48:43.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.924+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=RWLSSX6R84B7 tag=GZDJQPGWM3B2 type=write size=1594118 2023-12-03T11:48:43.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.926+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~T6BGQHBFOW80.0 size=284351 2023-12-03T11:48:43.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.926+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=T6BGQHBFOW80 tag=M3UTKO0ZB52O type=write size=1332927 2023-12-03T11:48:43.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.927+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=X79KPERM1DUP type=del 2023-12-03T11:48:43.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.929+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.0 size=1048576 2023-12-03T11:48:43.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.930+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.1 size=1048576 2023-12-03T11:48:43.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.931+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.2 size=970395 2023-12-03T11:48:43.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.931+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=0NX3H8VDZ5OK tag=YC7UTCZULMCX type=write size=4116123 2023-12-03T11:48:43.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.932+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=39JDSIL2MI91 type=del 2023-12-03T11:48:43.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.934+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~INMHMCSEJHIE.0 size=1048576 2023-12-03T11:48:43.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.935+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~INMHMCSEJHIE.1 size=1034417 2023-12-03T11:48:43.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.935+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=INMHMCSEJHIE tag=8JFYLBL3Z1JK type=write size=3131569 2023-12-03T11:48:43.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.935+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.0 size=1048576 2023-12-03T11:48:43.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.936+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.1 size=1048576 2023-12-03T11:48:43.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.936+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~KCGEXGK2THK8.2 size=837262 2023-12-03T11:48:43.940 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.938+0000 7f24574f5780 1 < canceled multipart upload key=RABUR02D2ZMD upload=KCGEXGK2THK8 tag=D0M9YCWZJ5JL type=write size=3982990 2023-12-03T11:48:43.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.940+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=EL73S15QXNB7 type=del size=2807487 2023-12-03T11:48:43.942 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.941+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=50NAT97RYU1F type=del 2023-12-03T11:48:43.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.942+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=S9WXL61PH5DE type=del size=258462 2023-12-03T11:48:43.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.943+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.0 size=1048576 2023-12-03T11:48:43.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.945+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.1 size=1048576 2023-12-03T11:48:43.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.947+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UG5EEM33C8VS.2 size=513892 2023-12-03T11:48:43.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.950+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=UG5EEM33C8VS tag=0HKSL3UZA3BO type=write size=3659620 2023-12-03T11:48:43.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.953+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~26LBWKCFGGAF.0 size=1048576 2023-12-03T11:48:43.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.954+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~26LBWKCFGGAF.1 size=397742 2023-12-03T11:48:43.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.954+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=26LBWKCFGGAF tag=51ZXUP0LPTV3 type=write size=2494894 2023-12-03T11:48:43.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.955+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~R81557V7I6IW.0 size=1048576 2023-12-03T11:48:43.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.955+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~R81557V7I6IW.1 size=152604 2023-12-03T11:48:43.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.957+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=R81557V7I6IW tag=STVSBWXPCMMI type=write size=2249756 2023-12-03T11:48:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.959+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.0 size=1048576 2023-12-03T11:48:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.960+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.1 size=1048576 2023-12-03T11:48:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.960+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.2 size=413630 2023-12-03T11:48:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.960+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=1T3451BHY9VR tag=QCO18STT7KEO type=write size=3559358 2023-12-03T11:48:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.962+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.0 size=1048576 2023-12-03T11:48:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.963+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.1 size=1048576 2023-12-03T11:48:43.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.963+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~JHT55G865J7C.2 size=876830 2023-12-03T11:48:43.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.965+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=JHT55G865J7C tag=Z4CDDQ1WUUQA type=write size=4022558 2023-12-03T11:48:43.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.967+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.0 size=1048576 2023-12-03T11:48:43.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.968+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.1 size=1048576 2023-12-03T11:48:43.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.969+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.2 size=263548 2023-12-03T11:48:43.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.969+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=FEUKV0UKO5Q0 tag=XAG2BBGVAGFQ type=write size=3409276 2023-12-03T11:48:43.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.971+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~W697I4L1N8WL.0 size=1048576 2023-12-03T11:48:43.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.971+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~W697I4L1N8WL.1 size=784737 2023-12-03T11:48:43.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.973+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=W697I4L1N8WL tag=VQ6NB5TM6I80 type=write size=2881889 2023-12-03T11:48:43.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.975+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=IQ8XHDSBI4IE type=del size=4160616 2023-12-03T11:48:43.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.976+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=913EQNWSIWGV type=del 2023-12-03T11:48:43.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.978+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~QI8VZF6CLEQP.0 size=1048576 2023-12-03T11:48:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.979+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~QI8VZF6CLEQP.1 size=911311 2023-12-03T11:48:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.979+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=QI8VZF6CLEQP tag=KMMXJ4X6L2C7 type=write size=3008463 2023-12-03T11:48:43.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.980+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=6ASGPSGIUZWH type=del 2023-12-03T11:48:43.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.981+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=9FNNGDR07GDE type=del size=1347545 2023-12-03T11:48:43.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.984+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=RSHZSP185KBR type=del size=1128617 2023-12-03T11:48:43.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.985+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=6MHZ6TNO7Q7B type=del size=907227 2023-12-03T11:48:43.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.986+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.0 size=1048576 2023-12-03T11:48:43.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.987+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.1 size=1048576 2023-12-03T11:48:43.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.988+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~W8H3CJLCXYD3.2 size=501582 2023-12-03T11:48:43.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.990+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=W8H3CJLCXYD3 tag=V3FA427JN8J4 type=write size=3647310 2023-12-03T11:48:43.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.991+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=SNW94YS1MZQL type=del size=3682643 2023-12-03T11:48:43.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.992+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.0 size=1048576 2023-12-03T11:48:43.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.993+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.1 size=1048576 2023-12-03T11:48:43.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.993+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~EHWT6PKZIGX9.2 size=308344 2023-12-03T11:48:43.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.995+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=EHWT6PKZIGX9 tag=4QRVBERE9JO9 type=write size=3454072 2023-12-03T11:48:43.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.996+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=MWE74FP9XWUN type=write size=170310 2023-12-03T11:48:43.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.997+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~Y3HNYZF3NTDR.0 size=740906 2023-12-03T11:48:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:43.999+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=Y3HNYZF3NTDR tag=KU2FOGW3QBB4 type=write size=1789482 2023-12-03T11:48:44.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.000+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=SV690IZLR48H type=del size=1443845 2023-12-03T11:48:44.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.002+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=16EP2U9NEW7L type=del 2023-12-03T11:48:44.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.003+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=74FOJ2NWSI5R type=del size=1243084 2023-12-03T11:48:44.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.004+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~T6BGQHBFOW80.0 size=284351 2023-12-03T11:48:44.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.006+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=T6BGQHBFOW80 tag=M3UTKO0ZB52O type=write size=1332927 2023-12-03T11:48:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.007+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=RT1GC41BPW2U type=del size=3708394 2023-12-03T11:48:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.009+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.0 size=1048576 2023-12-03T11:48:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.010+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.1 size=1048576 2023-12-03T11:48:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.010+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~0NX3H8VDZ5OK.2 size=970395 2023-12-03T11:48:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.012+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=0NX3H8VDZ5OK tag=YC7UTCZULMCX type=write size=4116123 2023-12-03T11:48:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.014+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~U1CZ9K07KWWN.0 size=1048576 2023-12-03T11:48:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.015+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~U1CZ9K07KWWN.1 size=735889 2023-12-03T11:48:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.015+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=U1CZ9K07KWWN tag=5B0ZJ37JWPZU type=write size=2833041 2023-12-03T11:48:44.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.016+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~INMHMCSEJHIE.0 size=1048576 2023-12-03T11:48:44.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.017+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~INMHMCSEJHIE.1 size=1034417 2023-12-03T11:48:44.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.019+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=INMHMCSEJHIE tag=8JFYLBL3Z1JK type=write size=3131569 2023-12-03T11:48:44.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.021+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~N43Y3DRX5L5W.0 size=1048576 2023-12-03T11:48:44.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.022+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~N43Y3DRX5L5W.1 size=82712 2023-12-03T11:48:44.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.022+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=N43Y3DRX5L5W tag=2589YSZ854WI type=write size=2179864 2023-12-03T11:48:44.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.023+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=EL73S15QXNB7 type=del 2023-12-03T11:48:44.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.025+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~04KZJYFC4MV9.0 size=1048576 2023-12-03T11:48:44.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.026+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~04KZJYFC4MV9.1 size=21215 2023-12-03T11:48:44.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.026+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=04KZJYFC4MV9 tag=CE2XRB2KHUVR type=write size=2118367 2023-12-03T11:48:44.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.027+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=S9WXL61PH5DE type=del 2023-12-03T11:48:44.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.029+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.0 size=1048576 2023-12-03T11:48:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.030+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.1 size=1048576 2023-12-03T11:48:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.031+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.2 size=283781 2023-12-03T11:48:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.031+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=KPH70OIWTZD8 tag=MKPRESNRMD9K type=write size=3429509 2023-12-03T11:48:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.032+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~26LBWKCFGGAF.0 size=1048576 2023-12-03T11:48:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.033+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~26LBWKCFGGAF.1 size=397742 2023-12-03T11:48:44.036 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.035+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=26LBWKCFGGAF tag=51ZXUP0LPTV3 type=write size=2494894 2023-12-03T11:48:44.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.037+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~1RXHJZVDREWN.0 size=171007 2023-12-03T11:48:44.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.037+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=1RXHJZVDREWN tag=8CL2V3TXAW49 type=write size=1219583 2023-12-03T11:48:44.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.039+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.0 size=1048576 2023-12-03T11:48:44.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.040+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.1 size=1048576 2023-12-03T11:48:44.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.041+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1T3451BHY9VR.2 size=413630 2023-12-03T11:48:44.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.043+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=1T3451BHY9VR tag=QCO18STT7KEO type=write size=3559358 2023-12-03T11:48:44.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.044+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=1NM6X9NIVS2M type=del size=3976477 2023-12-03T11:48:44.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.044+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.0 size=1048576 2023-12-03T11:48:44.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.045+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.1 size=1048576 2023-12-03T11:48:44.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.046+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~FEUKV0UKO5Q0.2 size=263548 2023-12-03T11:48:44.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.048+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=FEUKV0UKO5Q0 tag=XAG2BBGVAGFQ type=write size=3409276 2023-12-03T11:48:44.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.049+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=TXIY02IWRY1P type=write size=682666 2023-12-03T11:48:44.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.052+0000 7f24574f5780 1 < suggested update operation key=S0EH0Q0UA8HG tag=IQ8XHDSBI4IE type=del size=4160616 2023-12-03T11:48:44.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.055+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~IHDIGLKCOOS0.0 size=959253 2023-12-03T11:48:44.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.055+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=IHDIGLKCOOS0 tag=RMW9LP8GZTQA type=write size=2007829 2023-12-03T11:48:44.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.056+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~QI8VZF6CLEQP.0 size=1048576 2023-12-03T11:48:44.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.057+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~QI8VZF6CLEQP.1 size=911311 2023-12-03T11:48:44.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.059+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=QI8VZF6CLEQP tag=KMMXJ4X6L2C7 type=write size=3008463 2023-12-03T11:48:44.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.061+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~0RAPX9NH20YB.0 size=792688 2023-12-03T11:48:44.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.061+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=0RAPX9NH20YB tag=AXJOIBWGVHNK type=write size=1841264 2023-12-03T11:48:44.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.062+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=9FNNGDR07GDE type=del 2023-12-03T11:48:44.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.064+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~22VGYLEZ77C5.0 size=981696 2023-12-03T11:48:44.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.064+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=22VGYLEZ77C5 tag=59TITYIQ91FD type=write size=2030272 2023-12-03T11:48:44.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.065+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=6MHZ6TNO7Q7B type=del 2023-12-03T11:48:44.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.066+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=Q3PN0ENMMNI6 type=del size=1618903 2023-12-03T11:48:44.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.068+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=SNW94YS1MZQL type=del 2023-12-03T11:48:44.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.069+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=XNDEL8126JB1 type=write size=363805 2023-12-03T11:48:44.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.070+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=MWE74FP9XWUN type=write size=170310 2023-12-03T11:48:44.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.071+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=O1AYLJ3L3N40 type=del size=781341 2023-12-03T11:48:44.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.072+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=SV690IZLR48H type=del 2023-12-03T11:48:44.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.075+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~DKMLH8BE885H.0 size=1048576 2023-12-03T11:48:44.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.076+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~DKMLH8BE885H.1 size=704720 2023-12-03T11:48:44.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.076+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=DKMLH8BE885H tag=3AJP7OV5AMUM type=write size=2801872 2023-12-03T11:48:44.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.077+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=74FOJ2NWSI5R type=del 2023-12-03T11:48:44.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.078+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=WLNZUPGJFYKT type=del size=4038045 2023-12-03T11:48:44.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.080+0000 7f24574f5780 1 < suggested remove operation key=XRHVYVADRWET tag=RT1GC41BPW2U type=del 2023-12-03T11:48:44.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.082+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~0Q3DOM357CEL.0 size=818253 2023-12-03T11:48:44.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.082+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=0Q3DOM357CEL tag=FNVJCQA4QFW1 type=write size=1866829 2023-12-03T11:48:44.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.084+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~U1CZ9K07KWWN.0 size=1048576 2023-12-03T11:48:44.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.084+0000 7f24574f5780 1 < canceled part key=_multipart_XO42YZR60KQE.2~U1CZ9K07KWWN.1 size=735889 2023-12-03T11:48:44.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.086+0000 7f24574f5780 1 < canceled multipart upload key=XO42YZR60KQE upload=U1CZ9K07KWWN tag=5B0ZJ37JWPZU type=write size=2833041 2023-12-03T11:48:44.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.088+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.0 size=1048576 2023-12-03T11:48:44.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.088+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.1 size=1048576 2023-12-03T11:48:44.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.089+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.2 size=18391 2023-12-03T11:48:44.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.089+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=ITLJ3V7E1SE4 tag=0XJBQYTJY9DV type=write size=3164119 2023-12-03T11:48:44.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.090+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~N43Y3DRX5L5W.0 size=1048576 2023-12-03T11:48:44.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.091+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~N43Y3DRX5L5W.1 size=82712 2023-12-03T11:48:44.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.093+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=N43Y3DRX5L5W tag=2589YSZ854WI type=write size=2179864 2023-12-03T11:48:44.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.095+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~G8E6JNNDEEIO.0 size=1048576 2023-12-03T11:48:44.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.096+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~G8E6JNNDEEIO.1 size=587012 2023-12-03T11:48:44.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.096+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=G8E6JNNDEEIO tag=5BUE0YC8O4Q5 type=write size=2684164 2023-12-03T11:48:44.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.097+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~04KZJYFC4MV9.0 size=1048576 2023-12-03T11:48:44.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.098+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~04KZJYFC4MV9.1 size=21215 2023-12-03T11:48:44.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.100+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=04KZJYFC4MV9 tag=CE2XRB2KHUVR type=write size=2118367 2023-12-03T11:48:44.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.101+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=MYD9GRLUZ9OH type=del size=1903524 2023-12-03T11:48:44.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.102+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.0 size=1048576 2023-12-03T11:48:44.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.103+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.1 size=1048576 2023-12-03T11:48:44.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.103+0000 7f24574f5780 1 < canceled part key=_multipart_XRHVYVADRWET.2~KPH70OIWTZD8.2 size=283781 2023-12-03T11:48:44.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.105+0000 7f24574f5780 1 < canceled multipart upload key=XRHVYVADRWET upload=KPH70OIWTZD8 tag=MKPRESNRMD9K type=write size=3429509 2023-12-03T11:48:44.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.106+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=2IU7572ST94G type=del size=3324906 2023-12-03T11:48:44.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.107+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~1RXHJZVDREWN.0 size=171007 2023-12-03T11:48:44.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.110+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=1RXHJZVDREWN tag=8CL2V3TXAW49 type=write size=1219583 2023-12-03T11:48:44.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.112+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~2SCBASP2PMC9.0 size=717009 2023-12-03T11:48:44.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.112+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=2SCBASP2PMC9 tag=9XNA9Z4CM9A0 type=write size=1765585 2023-12-03T11:48:44.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.113+0000 7f24574f5780 1 < canceled operation key=SUBILSL92D96 tag=1NM6X9NIVS2M type=del size=3976477 2023-12-03T11:48:44.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.114+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=1YBPODFKKG7Z type=del size=893247 2023-12-03T11:48:44.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.115+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=TXIY02IWRY1P type=write size=682666 2023-12-03T11:48:44.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.117+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=BFWBRFKKO8VU type=del size=90954 2023-12-03T11:48:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.118+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~IHDIGLKCOOS0.0 size=959253 2023-12-03T11:48:44.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.120+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=IHDIGLKCOOS0 tag=RMW9LP8GZTQA type=write size=2007829 2023-12-03T11:48:44.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.121+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=HF63O3TZ38Z2 type=del size=4081996 2023-12-03T11:48:44.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.121+0000 7f24574f5780 1 < canceled part key=_multipart_LS4OEAGZWQXC.2~0RAPX9NH20YB.0 size=792688 2023-12-03T11:48:44.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.125+0000 7f24574f5780 1 < canceled multipart upload key=LS4OEAGZWQXC upload=0RAPX9NH20YB tag=AXJOIBWGVHNK type=write size=1841264 2023-12-03T11:48:44.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.126+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=CHO5E28S3IWM type=del size=4130411 2023-12-03T11:48:44.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.127+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~22VGYLEZ77C5.0 size=981696 2023-12-03T11:48:44.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.129+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=22VGYLEZ77C5 tag=59TITYIQ91FD type=write size=2030272 2023-12-03T11:48:44.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.130+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=38XRJ66GNBDX type=write size=718904 2023-12-03T11:48:44.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.131+0000 7f24574f5780 1 < canceled operation key=PS1OYWI44L80 tag=Q3PN0ENMMNI6 type=del size=1618903 2023-12-03T11:48:44.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.133+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~A3IL92TQOVYX.0 size=1048576 2023-12-03T11:48:44.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.134+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~A3IL92TQOVYX.1 size=541449 2023-12-03T11:48:44.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.134+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=A3IL92TQOVYX tag=C5IIGCBD33CK type=write size=2638601 2023-12-03T11:48:44.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.136+0000 7f24574f5780 1 < completed write operation key=1V39QA10M1HY tag=XNDEL8126JB1 type=write size=363805 2023-12-03T11:48:44.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.137+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=LEHPE1939JFF type=del size=400383 2023-12-03T11:48:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.139+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=O1AYLJ3L3N40 type=del 2023-12-03T11:48:44.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.140+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=RYQXMBQNFLOX type=del size=842450 2023-12-03T11:48:44.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.141+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~DKMLH8BE885H.0 size=1048576 2023-12-03T11:48:44.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.142+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~DKMLH8BE885H.1 size=704720 2023-12-03T11:48:44.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.144+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=DKMLH8BE885H tag=3AJP7OV5AMUM type=write size=2801872 2023-12-03T11:48:44.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.146+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~ZTGO5MEGF0RK.0 size=1048576 2023-12-03T11:48:44.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.147+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~ZTGO5MEGF0RK.1 size=106150 2023-12-03T11:48:44.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.147+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=ZTGO5MEGF0RK tag=DPD0AWUABDPT type=write size=2203302 2023-12-03T11:48:44.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.149+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=WLNZUPGJFYKT type=del 2023-12-03T11:48:44.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.151+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.0 size=1048576 2023-12-03T11:48:44.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.152+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.1 size=1048576 2023-12-03T11:48:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.152+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.2 size=449971 2023-12-03T11:48:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.152+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=LABV2OLZHZD3 tag=323EPLDKAHRQ type=write size=3595699 2023-12-03T11:48:44.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.154+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~0Q3DOM357CEL.0 size=818253 2023-12-03T11:48:44.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.156+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=0Q3DOM357CEL tag=FNVJCQA4QFW1 type=write size=1866829 2023-12-03T11:48:44.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.158+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~R6ITOE3IJN8G.0 size=767620 2023-12-03T11:48:44.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.158+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=R6ITOE3IJN8G tag=X3HA3V3KOWXQ type=write size=1816196 2023-12-03T11:48:44.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.159+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.0 size=1048576 2023-12-03T11:48:44.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.160+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.1 size=1048576 2023-12-03T11:48:44.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.162+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~ITLJ3V7E1SE4.2 size=18391 2023-12-03T11:48:44.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.164+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=ITLJ3V7E1SE4 tag=0XJBQYTJY9DV type=write size=3164119 2023-12-03T11:48:44.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.166+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~MFR3D4980FIK.0 size=1048576 2023-12-03T11:48:44.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.167+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~MFR3D4980FIK.1 size=12747 2023-12-03T11:48:44.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.167+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=MFR3D4980FIK tag=OZSKW4PI0ELS type=write size=2109899 2023-12-03T11:48:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.167+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~G8E6JNNDEEIO.0 size=1048576 2023-12-03T11:48:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.168+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~G8E6JNNDEEIO.1 size=587012 2023-12-03T11:48:44.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.170+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=G8E6JNNDEEIO tag=5BUE0YC8O4Q5 type=write size=2684164 2023-12-03T11:48:44.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.172+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~VJVMO5CAVW1L.0 size=1048576 2023-12-03T11:48:44.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.173+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~VJVMO5CAVW1L.1 size=793744 2023-12-03T11:48:44.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.173+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=VJVMO5CAVW1L tag=MW407L6YUURO type=write size=2890896 2023-12-03T11:48:44.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.175+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=MYD9GRLUZ9OH type=del 2023-12-03T11:48:44.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.177+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~ZMTA9NEFLKY8.0 size=785085 2023-12-03T11:48:44.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.177+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=ZMTA9NEFLKY8 tag=373W7V18G615 type=write size=1833661 2023-12-03T11:48:44.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.178+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=2IU7572ST94G type=del 2023-12-03T11:48:44.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.179+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=ZAUE7XXEQNH1 type=del size=561530 2023-12-03T11:48:44.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.180+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~2SCBASP2PMC9.0 size=717009 2023-12-03T11:48:44.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.182+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=2SCBASP2PMC9 tag=9XNA9Z4CM9A0 type=write size=1765585 2023-12-03T11:48:44.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.184+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=XOCADU605C7R type=del size=3708863 2023-12-03T11:48:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.185+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=1YBPODFKKG7Z type=del 2023-12-03T11:48:44.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.186+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=8V6LY1SCBYDT type=del size=859596 2023-12-03T11:48:44.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.187+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=BFWBRFKKO8VU type=del 2023-12-03T11:48:44.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.189+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=6MMSGL3ENR4E type=del size=788759 2023-12-03T11:48:44.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.190+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=HF63O3TZ38Z2 type=del 2023-12-03T11:48:44.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.191+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=4VC4B901JCMZ type=del size=65198 2023-12-03T11:48:44.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.197+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=CHO5E28S3IWM type=del 2023-12-03T11:48:44.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.201+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=F4GXKR1EMH4L type=write size=531316 2023-12-03T11:48:44.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.202+0000 7f24574f5780 1 < completed write operation key=ON2E6H35N9FJ tag=38XRJ66GNBDX type=write size=718904 2023-12-03T11:48:44.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.204+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.0 size=1048576 2023-12-03T11:48:44.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.205+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.1 size=1048576 2023-12-03T11:48:44.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.206+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.2 size=70348 2023-12-03T11:48:44.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.206+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=AA9G9RPHN9UY tag=ZD9YJZHQO1FM type=write size=3216076 2023-12-03T11:48:44.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.207+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~A3IL92TQOVYX.0 size=1048576 2023-12-03T11:48:44.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.208+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~A3IL92TQOVYX.1 size=541449 2023-12-03T11:48:44.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.211+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=A3IL92TQOVYX tag=C5IIGCBD33CK type=write size=2638601 2023-12-03T11:48:44.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.212+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=8XYDG3S2RHMG type=del size=1436200 2023-12-03T11:48:44.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.213+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=LEHPE1939JFF type=del 2023-12-03T11:48:44.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.214+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=T8Q86BCD7RN0 type=del size=3251024 2023-12-03T11:48:44.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.215+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=RYQXMBQNFLOX type=del 2023-12-03T11:48:44.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.218+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.0 size=1048576 2023-12-03T11:48:44.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.219+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.1 size=1048576 2023-12-03T11:48:44.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.220+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.2 size=681849 2023-12-03T11:48:44.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.220+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=OTVDSDVYMVF9 tag=LPZVGR7RT3FW type=write size=3827577 2023-12-03T11:48:44.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.221+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~ZTGO5MEGF0RK.0 size=1048576 2023-12-03T11:48:44.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.222+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~ZTGO5MEGF0RK.1 size=106150 2023-12-03T11:48:44.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.224+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=ZTGO5MEGF0RK tag=DPD0AWUABDPT type=write size=2203302 2023-12-03T11:48:44.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.225+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=HSBL46BXCF2Q type=del size=84715 2023-12-03T11:48:44.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.226+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.0 size=1048576 2023-12-03T11:48:44.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.227+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.1 size=1048576 2023-12-03T11:48:44.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.229+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~LABV2OLZHZD3.2 size=449971 2023-12-03T11:48:44.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.231+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=LABV2OLZHZD3 tag=323EPLDKAHRQ type=write size=3595699 2023-12-03T11:48:44.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.232+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=UDB2339PLH78 type=del size=693062 2023-12-03T11:48:44.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.233+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~R6ITOE3IJN8G.0 size=767620 2023-12-03T11:48:44.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.235+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=R6ITOE3IJN8G tag=X3HA3V3KOWXQ type=write size=1816196 2023-12-03T11:48:44.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.237+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~O34EFKRPF0EE.0 size=1048576 2023-12-03T11:48:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.238+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~O34EFKRPF0EE.1 size=33637 2023-12-03T11:48:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.238+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=O34EFKRPF0EE tag=TY684H4486K1 type=write size=2130789 2023-12-03T11:48:44.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.239+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~MFR3D4980FIK.0 size=1048576 2023-12-03T11:48:44.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.240+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~MFR3D4980FIK.1 size=12747 2023-12-03T11:48:44.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.243+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=MFR3D4980FIK tag=OZSKW4PI0ELS type=write size=2109899 2023-12-03T11:48:44.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.244+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=B7RNA6LSGQVS type=del size=1649588 2023-12-03T11:48:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.245+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~VJVMO5CAVW1L.0 size=1048576 2023-12-03T11:48:44.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.246+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~VJVMO5CAVW1L.1 size=793744 2023-12-03T11:48:44.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.249+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=VJVMO5CAVW1L tag=MW407L6YUURO type=write size=2890896 2023-12-03T11:48:44.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.251+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.0 size=1048576 2023-12-03T11:48:44.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.252+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.1 size=1048576 2023-12-03T11:48:44.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.253+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.2 size=897311 2023-12-03T11:48:44.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.253+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=R4EI3DLC4ODX tag=TD8L4LGF5ICD type=write size=4043039 2023-12-03T11:48:44.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.255+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~ZMTA9NEFLKY8.0 size=785085 2023-12-03T11:48:44.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.257+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=ZMTA9NEFLKY8 tag=373W7V18G615 type=write size=1833661 2023-12-03T11:48:44.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.259+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.0 size=1048576 2023-12-03T11:48:44.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.260+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.1 size=1048576 2023-12-03T11:48:44.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.261+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.2 size=813448 2023-12-03T11:48:44.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.261+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=B1FNH3W28YAI tag=1U5P17YAFHXV type=write size=3959176 2023-12-03T11:48:44.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.263+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=ZAUE7XXEQNH1 type=del 2023-12-03T11:48:44.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.264+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=O6HCZW1UFTR7 type=del size=2752239 2023-12-03T11:48:44.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.265+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=XOCADU605C7R type=del 2023-12-03T11:48:44.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.267+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=JSQ7UZ8IC5KC type=del size=209480 2023-12-03T11:48:44.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.268+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=8V6LY1SCBYDT type=del 2023-12-03T11:48:44.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.269+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=039J6T9WYFCD type=del size=3474030 2023-12-03T11:48:44.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.270+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=6MMSGL3ENR4E type=del 2023-12-03T11:48:44.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.271+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=AM71F2Q67Y38 type=del size=4135168 2023-12-03T11:48:44.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.272+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=4VC4B901JCMZ type=del 2023-12-03T11:48:44.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.274+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~TDAAIWLQ7XIB.0 size=667775 2023-12-03T11:48:44.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.274+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=TDAAIWLQ7XIB tag=3QAXDGHM4MDU type=write size=1716351 2023-12-03T11:48:44.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.277+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=F4GXKR1EMH4L type=write size=531316 2023-12-03T11:48:44.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.278+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=EKT0XTU7DMYF type=del size=522192 2023-12-03T11:48:44.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.279+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.0 size=1048576 2023-12-03T11:48:44.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.280+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.1 size=1048576 2023-12-03T11:48:44.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.281+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~AA9G9RPHN9UY.2 size=70348 2023-12-03T11:48:44.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.285+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=AA9G9RPHN9UY tag=ZD9YJZHQO1FM type=write size=3216076 2023-12-03T11:48:44.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.288+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.0 size=1048576 2023-12-03T11:48:44.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.288+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.1 size=1048576 2023-12-03T11:48:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.289+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.2 size=868664 2023-12-03T11:48:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.289+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=RQD5NVEG4ZCG tag=YK6E0S2NKALW type=write size=4014392 2023-12-03T11:48:44.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.292+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=8XYDG3S2RHMG type=del 2023-12-03T11:48:44.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.294+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.0 size=1048576 2023-12-03T11:48:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.295+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.1 size=1048576 2023-12-03T11:48:44.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.295+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.2 size=516430 2023-12-03T11:48:44.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.295+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=NX4KT24DRH96 tag=636382EXCK3Z type=write size=3662158 2023-12-03T11:48:44.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.297+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=T8Q86BCD7RN0 type=del 2023-12-03T11:48:44.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.298+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=A5W0E79QHGB8 type=del size=3432280 2023-12-03T11:48:44.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.299+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.0 size=1048576 2023-12-03T11:48:44.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.300+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.1 size=1048576 2023-12-03T11:48:44.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.301+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~OTVDSDVYMVF9.2 size=681849 2023-12-03T11:48:44.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.304+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=OTVDSDVYMVF9 tag=LPZVGR7RT3FW type=write size=3827577 2023-12-03T11:48:44.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.305+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=6UF49N7NZZGN type=del size=3459129 2023-12-03T11:48:44.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.306+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=HSBL46BXCF2Q type=del size=84715 2023-12-03T11:48:44.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.308+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~5WWR0YPHQNOX.0 size=1048576 2023-12-03T11:48:44.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.309+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~5WWR0YPHQNOX.1 size=646323 2023-12-03T11:48:44.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.309+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=5WWR0YPHQNOX tag=O99VWALDX3WX type=write size=2743475 2023-12-03T11:48:44.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.310+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=UDB2339PLH78 type=del 2023-12-03T11:48:44.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.313+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FC2JD9IVMA57.0 size=1048576 2023-12-03T11:48:44.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.314+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FC2JD9IVMA57.1 size=166326 2023-12-03T11:48:44.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.314+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=FC2JD9IVMA57 tag=5ME5BIXR4AV9 type=write size=2263478 2023-12-03T11:48:44.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.315+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~O34EFKRPF0EE.0 size=1048576 2023-12-03T11:48:44.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.315+0000 7f24574f5780 1 < canceled part key=_multipart_6WY91DFIRUN8.2~O34EFKRPF0EE.1 size=33637 2023-12-03T11:48:44.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.317+0000 7f24574f5780 1 < canceled multipart upload key=6WY91DFIRUN8 upload=O34EFKRPF0EE tag=TY684H4486K1 type=write size=2130789 2023-12-03T11:48:44.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.319+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~68EBWTBXRDJL.0 size=1048576 2023-12-03T11:48:44.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.320+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~68EBWTBXRDJL.1 size=10734 2023-12-03T11:48:44.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.320+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=68EBWTBXRDJL tag=XNZL1NQ66M4G type=write size=2107886 2023-12-03T11:48:44.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.321+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=B7RNA6LSGQVS type=del size=1649588 2023-12-03T11:48:44.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.322+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=ZDC31E8CP3BU type=del size=2096754 2023-12-03T11:48:44.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.323+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.0 size=1048576 2023-12-03T11:48:44.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.324+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.1 size=1048576 2023-12-03T11:48:44.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.325+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~R4EI3DLC4ODX.2 size=897311 2023-12-03T11:48:44.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.327+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=R4EI3DLC4ODX tag=TD8L4LGF5ICD type=write size=4043039 2023-12-03T11:48:44.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.328+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=HINX49MNLLH1 type=del size=3161339 2023-12-03T11:48:44.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.328+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.0 size=1048576 2023-12-03T11:48:44.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.329+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.1 size=1048576 2023-12-03T11:48:44.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.330+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~B1FNH3W28YAI.2 size=813448 2023-12-03T11:48:44.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.332+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=B1FNH3W28YAI tag=1U5P17YAFHXV type=write size=3959176 2023-12-03T11:48:44.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.333+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=NPUFMDLMFX0O type=write size=337848 2023-12-03T11:48:44.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.336+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=O6HCZW1UFTR7 type=del size=2752239 2023-12-03T11:48:44.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.338+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1R5GY9BLGU73.0 size=1048576 2023-12-03T11:48:44.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.339+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1R5GY9BLGU73.1 size=691444 2023-12-03T11:48:44.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.339+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=1R5GY9BLGU73 tag=R8Y4NGHWDRZ1 type=write size=2788596 2023-12-03T11:48:44.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.340+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=JSQ7UZ8IC5KC type=del 2023-12-03T11:48:44.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.341+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=AXPHKAEBXRLO type=del size=3252262 2023-12-03T11:48:44.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.342+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=039J6T9WYFCD type=del 2023-12-03T11:48:44.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.343+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=CYG2ZV5LRNM1 type=del size=3764632 2023-12-03T11:48:44.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.344+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=AM71F2Q67Y38 type=del 2023-12-03T11:48:44.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.346+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=3RC53HWDBTQB type=del size=3933306 2023-12-03T11:48:44.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.347+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~TDAAIWLQ7XIB.0 size=667775 2023-12-03T11:48:44.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.349+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=TDAAIWLQ7XIB tag=3QAXDGHM4MDU type=write size=1716351 2023-12-03T11:48:44.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.350+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=UJ5HF31FYTX7 type=del size=1143449 2023-12-03T11:48:44.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.351+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=EKT0XTU7DMYF type=del 2023-12-03T11:48:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.352+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=Z94FPRV01TSN type=write size=978943 2023-12-03T11:48:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.353+0000 7f24574f5780 1 < canceled part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.0 size=1048576 2023-12-03T11:48:44.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.354+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.1 size=1048576 2023-12-03T11:48:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.355+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~RQD5NVEG4ZCG.2 size=868664 2023-12-03T11:48:44.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.357+0000 7f24574f5780 1 < canceled multipart upload key=WD7U31FW3NGZ upload=RQD5NVEG4ZCG tag=YK6E0S2NKALW type=write size=4014392 2023-12-03T11:48:44.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.358+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=4H02QZSNOJDP type=del size=370023 2023-12-03T11:48:44.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.359+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.0 size=1048576 2023-12-03T11:48:44.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.359+0000 7f24574f5780 1 < canceled part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.1 size=1048576 2023-12-03T11:48:44.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.360+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~NX4KT24DRH96.2 size=516430 2023-12-03T11:48:44.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.362+0000 7f24574f5780 1 < canceled multipart upload key=HM9JHOV8W24F upload=NX4KT24DRH96 tag=636382EXCK3Z type=write size=3662158 2023-12-03T11:48:44.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.364+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.0 size=1048576 2023-12-03T11:48:44.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.365+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.1 size=1048576 2023-12-03T11:48:44.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.366+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.2 size=680795 2023-12-03T11:48:44.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.366+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=PZUGZM0OD1R1 tag=7D1R583JVXAE type=write size=3826523 2023-12-03T11:48:44.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.367+0000 7f24574f5780 -1 < failed to complete operation key=RABUR02D2ZMD tag=A5W0E79QHGB8 type=del size=3432280: (22) Invalid argument 2023-12-03T11:48:44.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.368+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=UGZI2O09T7YL type=del size=2279303 2023-12-03T11:48:44.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.369+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=6UF49N7NZZGN type=del 2023-12-03T11:48:44.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.371+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=NJDEUK6VHCM5 type=write size=525961 2023-12-03T11:48:44.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.372+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~5WWR0YPHQNOX.0 size=1048576 2023-12-03T11:48:44.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.372+0000 7f24574f5780 1 < canceled part key=_multipart_3EOEF9UMATIE.2~5WWR0YPHQNOX.1 size=646323 2023-12-03T11:48:44.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.374+0000 7f24574f5780 1 < canceled multipart upload key=3EOEF9UMATIE upload=5WWR0YPHQNOX tag=O99VWALDX3WX type=write size=2743475 2023-12-03T11:48:44.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.375+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=AJNYFLRFBCR8 type=del size=3524684 2023-12-03T11:48:44.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.376+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FC2JD9IVMA57.0 size=1048576 2023-12-03T11:48:44.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.377+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FC2JD9IVMA57.1 size=166326 2023-12-03T11:48:44.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.379+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=FC2JD9IVMA57 tag=5ME5BIXR4AV9 type=write size=2263478 2023-12-03T11:48:44.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.380+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=VMIX22Q6AP96 type=del size=3179478 2023-12-03T11:48:44.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.381+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~68EBWTBXRDJL.0 size=1048576 2023-12-03T11:48:44.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.382+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~68EBWTBXRDJL.1 size=10734 2023-12-03T11:48:44.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.384+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=68EBWTBXRDJL tag=XNZL1NQ66M4G type=write size=2107886 2023-12-03T11:48:44.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.386+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.0 size=1048576 2023-12-03T11:48:44.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.386+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.1 size=1048576 2023-12-03T11:48:44.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.387+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.2 size=957011 2023-12-03T11:48:44.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.387+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=895BJDPVDKNT tag=NZ0LL3ZM3PML type=write size=4102739 2023-12-03T11:48:44.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.389+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=ZDC31E8CP3BU type=del 2023-12-03T11:48:44.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.390+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=1AMWQ5492U02 type=write size=303872 2023-12-03T11:48:44.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.393+0000 7f24574f5780 1 < suggested remove operation key=9LKFY1U0JRBQ tag=HINX49MNLLH1 type=del 2023-12-03T11:48:44.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.394+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=3ZAXQI6FZU6L type=del size=3786599 2023-12-03T11:48:44.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.395+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=NPUFMDLMFX0O type=write size=337848 2023-12-03T11:48:44.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.396+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=VA65SS0SJYDF type=del size=2354827 2023-12-03T11:48:44.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.398+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1R5GY9BLGU73.0 size=1048576 2023-12-03T11:48:44.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.399+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1R5GY9BLGU73.1 size=691444 2023-12-03T11:48:44.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.401+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=1R5GY9BLGU73 tag=R8Y4NGHWDRZ1 type=write size=2788596 2023-12-03T11:48:44.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.402+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=U73MLBGWHT6T type=del size=3679475 2023-12-03T11:48:44.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.404+0000 7f24574f5780 1 < suggested remove operation key=1V39QA10M1HY tag=AXPHKAEBXRLO type=del 2023-12-03T11:48:44.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.406+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.0 size=1048576 2023-12-03T11:48:44.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.407+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.1 size=1048576 2023-12-03T11:48:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.408+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.2 size=485481 2023-12-03T11:48:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.408+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=3JQUC2KIHMJY tag=MYUFQ2QVL44Q type=write size=3631209 2023-12-03T11:48:44.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.410+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=CYG2ZV5LRNM1 type=del 2023-12-03T11:48:44.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.411+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=LEDXZS90T0Z8 type=del size=742296 2023-12-03T11:48:44.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.412+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=3RC53HWDBTQB type=del size=3933306: (22) Invalid argument 2023-12-03T11:48:44.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.413+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=IQ16PRX100JD type=del size=2783521 2023-12-03T11:48:44.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.414+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=UJ5HF31FYTX7 type=del size=1143449 2023-12-03T11:48:44.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.415+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=XOBSEU6A4IQI type=del size=1465636 2023-12-03T11:48:44.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.419+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=Z94FPRV01TSN type=write size=978943 2023-12-03T11:48:44.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.421+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~WGMNPY9HJ4Q0.0 size=1048576 2023-12-03T11:48:44.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.423+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~WGMNPY9HJ4Q0.1 size=263755 2023-12-03T11:48:44.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.423+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=WGMNPY9HJ4Q0 tag=CQAALYKK9U2E type=write size=2360907 2023-12-03T11:48:44.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.424+0000 7f24574f5780 -1 < failed to complete operation key=9LKFY1U0JRBQ tag=4H02QZSNOJDP type=del size=370023: (22) Invalid argument 2023-12-03T11:48:44.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.425+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=DRGWKVQ8E7NM type=write size=539153 2023-12-03T11:48:44.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.426+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.0 size=1048576 2023-12-03T11:48:44.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.427+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.1 size=1048576 2023-12-03T11:48:44.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.428+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~PZUGZM0OD1R1.2 size=680795 2023-12-03T11:48:44.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.430+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=PZUGZM0OD1R1 tag=7D1R583JVXAE type=write size=3826523 2023-12-03T11:48:44.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.431+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=3T11FO3T2OZL type=del size=3771795 2023-12-03T11:48:44.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.432+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=UGZI2O09T7YL type=del 2023-12-03T11:48:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.433+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=KOZAQAPP4UBQ type=del size=3311847 2023-12-03T11:48:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.435+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=NJDEUK6VHCM5 type=write size=525961 2023-12-03T11:48:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.436+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=RHUE28WKOT1F type=del size=1923408 2023-12-03T11:48:44.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.438+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=AJNYFLRFBCR8 type=del 2023-12-03T11:48:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.439+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=MUFQC5BC6X2Y type=del size=1730267 2023-12-03T11:48:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.440+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=VMIX22Q6AP96 type=del 2023-12-03T11:48:44.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.442+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~N69258VYY9XP.0 size=1048576 2023-12-03T11:48:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.443+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~N69258VYY9XP.1 size=343297 2023-12-03T11:48:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.443+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=N69258VYY9XP tag=B8QZ7MUCN6RC type=write size=2440449 2023-12-03T11:48:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.444+0000 7f24574f5780 1 < canceled part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.0 size=1048576 2023-12-03T11:48:44.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.445+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.1 size=1048576 2023-12-03T11:48:44.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.445+0000 7f24574f5780 1 < canceled part key=_multipart_3EOEF9UMATIE.2~895BJDPVDKNT.2 size=957011 2023-12-03T11:48:44.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.447+0000 7f24574f5780 1 < canceled multipart upload key=3EOEF9UMATIE upload=895BJDPVDKNT tag=NZ0LL3ZM3PML type=write size=4102739 2023-12-03T11:48:44.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.448+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=5MGE50HVYH58 type=del size=2142107 2023-12-03T11:48:44.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.449+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=1AMWQ5492U02 type=write size=303872 2023-12-03T11:48:44.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.451+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~KQCRLKYFKC2X.0 size=730631 2023-12-03T11:48:44.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.451+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=KQCRLKYFKC2X tag=C7LCE5H1K866 type=write size=1779207 2023-12-03T11:48:44.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.452+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=3ZAXQI6FZU6L type=del 2023-12-03T11:48:44.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.454+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~M3WIU10XTMJJ.0 size=872840 2023-12-03T11:48:44.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.454+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=M3WIU10XTMJJ tag=0M9V8ZQWO7DL type=write size=1921416 2023-12-03T11:48:44.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.456+0000 7f24574f5780 1 < suggested remove operation key=SUBILSL92D96 tag=VA65SS0SJYDF type=del 2023-12-03T11:48:44.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.457+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=GC0NMTGNQ40G type=del size=3153677 2023-12-03T11:48:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.458+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=U73MLBGWHT6T type=del 2023-12-03T11:48:44.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.459+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=A0Q05ZFRJR6S type=del size=3682385 2023-12-03T11:48:44.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.461+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.0 size=1048576 2023-12-03T11:48:44.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.461+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.1 size=1048576 2023-12-03T11:48:44.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.462+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~3JQUC2KIHMJY.2 size=485481 2023-12-03T11:48:44.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.464+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=3JQUC2KIHMJY tag=MYUFQ2QVL44Q type=write size=3631209 2023-12-03T11:48:44.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.466+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=TM0F7FTIMY4L type=del size=3331104 2023-12-03T11:48:44.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.467+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=LEDXZS90T0Z8 type=del 2023-12-03T11:48:44.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.468+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=PX1E6JN92948 type=del size=3461896 2023-12-03T11:48:44.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.469+0000 7f24574f5780 1 < canceled operation key=6WY91DFIRUN8 tag=IQ16PRX100JD type=del size=2783521 2023-12-03T11:48:44.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.470+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=JHSOUF3HL9T9 type=del size=2401643 2023-12-03T11:48:44.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.471+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=XOBSEU6A4IQI type=del 2023-12-03T11:48:44.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.473+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=MI03X5B1MMCQ type=del size=3334355 2023-12-03T11:48:44.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.474+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~WGMNPY9HJ4Q0.0 size=1048576 2023-12-03T11:48:44.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.475+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~WGMNPY9HJ4Q0.1 size=263755 2023-12-03T11:48:44.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.478+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=WGMNPY9HJ4Q0 tag=CQAALYKK9U2E type=write size=2360907 2023-12-03T11:48:44.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.479+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=XFS3XQRCS65A type=del size=1973684 2023-12-03T11:48:44.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.481+0000 7f24574f5780 1 < completed write operation key=1V1C59DP8YPF tag=DRGWKVQ8E7NM type=write size=539153 2023-12-03T11:48:44.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.482+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=D64134JLCUD4 type=del size=357294 2023-12-03T11:48:44.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.483+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=3T11FO3T2OZL type=del 2023-12-03T11:48:44.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.485+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~9Z2JUKQERD4I.0 size=1048576 2023-12-03T11:48:44.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.486+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~9Z2JUKQERD4I.1 size=244864 2023-12-03T11:48:44.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.486+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=9Z2JUKQERD4I tag=SOH1OOQSBF5K type=write size=2342016 2023-12-03T11:48:44.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.487+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=KOZAQAPP4UBQ type=del 2023-12-03T11:48:44.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.488+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=65CEUV5PD4E9 type=del size=3848548 2023-12-03T11:48:44.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.489+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=RHUE28WKOT1F type=del 2023-12-03T11:48:44.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.491+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=1AYPU5E0S5SS type=write size=987087 2023-12-03T11:48:44.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.492+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=MUFQC5BC6X2Y type=del 2023-12-03T11:48:44.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.493+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=O7WDAOJ8GTF6 type=del size=2581842 2023-12-03T11:48:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.494+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~N69258VYY9XP.0 size=1048576 2023-12-03T11:48:44.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.495+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~N69258VYY9XP.1 size=343297 2023-12-03T11:48:44.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.497+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=N69258VYY9XP tag=B8QZ7MUCN6RC type=write size=2440449 2023-12-03T11:48:44.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.499+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~LODZST3HBZRX.0 size=1048576 2023-12-03T11:48:44.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.500+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~LODZST3HBZRX.1 size=438588 2023-12-03T11:48:44.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.500+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=LODZST3HBZRX tag=YMC5OA1G1ZLJ type=write size=2535740 2023-12-03T11:48:44.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.502+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=5MGE50HVYH58 type=del size=2142107 2023-12-03T11:48:44.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.503+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=S10V5F8E5F53 type=del size=902365 2023-12-03T11:48:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.505+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~KQCRLKYFKC2X.0 size=730631 2023-12-03T11:48:44.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.508+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=KQCRLKYFKC2X tag=C7LCE5H1K866 type=write size=1779207 2023-12-03T11:48:44.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.509+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=SGHKESXAIP0F type=del size=1680050 2023-12-03T11:48:44.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.510+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~M3WIU10XTMJJ.0 size=872840 2023-12-03T11:48:44.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.512+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=M3WIU10XTMJJ tag=0M9V8ZQWO7DL type=write size=1921416 2023-12-03T11:48:44.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.514+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.0 size=1048576 2023-12-03T11:48:44.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.515+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.1 size=1048576 2023-12-03T11:48:44.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.516+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.2 size=630503 2023-12-03T11:48:44.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.516+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=LLLCPIIQ4WXH tag=AG9M5412JWPK type=write size=3776231 2023-12-03T11:48:44.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.518+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=GC0NMTGNQ40G type=del size=3153677 2023-12-03T11:48:44.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.519+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=NKXVHG42JDG2 type=del size=2799479 2023-12-03T11:48:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.520+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=A0Q05ZFRJR6S type=del 2023-12-03T11:48:44.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.521+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=FKVRVD3K7ONM type=del size=314398 2023-12-03T11:48:44.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.523+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=TM0F7FTIMY4L type=del 2023-12-03T11:48:44.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.524+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=KYU91UK973EX type=del size=3967337 2023-12-03T11:48:44.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.525+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=PX1E6JN92948 type=del 2023-12-03T11:48:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.526+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=J3ZOBUO14X65 type=del size=1764400 2023-12-03T11:48:44.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.527+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=JHSOUF3HL9T9 type=del 2023-12-03T11:48:44.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.528+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=BPUW6A8RIY0O type=write size=448089 2023-12-03T11:48:44.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.529+0000 7f24574f5780 1 < canceled operation key=BV8C0VSJ1GTW tag=MI03X5B1MMCQ type=del size=3334355 2023-12-03T11:48:44.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.530+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=KNK1E7LZEQ4M type=write size=207888 2023-12-03T11:48:44.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.532+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=XFS3XQRCS65A type=del 2023-12-03T11:48:44.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.533+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~B5BKK6I40RXD.0 size=231561 2023-12-03T11:48:44.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.533+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=B5BKK6I40RXD tag=UJNJ16ZS5WJ1 type=write size=1280137 2023-12-03T11:48:44.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.535+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=D64134JLCUD4 type=del size=357294: (22) Invalid argument 2023-12-03T11:48:44.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.536+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=ZHGPIHHBRVZI type=del size=2937066 2023-12-03T11:48:44.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.537+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~9Z2JUKQERD4I.0 size=1048576 2023-12-03T11:48:44.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.538+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~9Z2JUKQERD4I.1 size=244864 2023-12-03T11:48:44.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.540+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=9Z2JUKQERD4I tag=SOH1OOQSBF5K type=write size=2342016 2023-12-03T11:48:44.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.541+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=MHR0N5EOPJ1K type=del size=3940029 2023-12-03T11:48:44.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.543+0000 7f24574f5780 1 < suggested remove operation key=LS4OEAGZWQXC tag=65CEUV5PD4E9 type=del 2023-12-03T11:48:44.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.544+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=1SOQMITDDXS1 type=del size=3774764 2023-12-03T11:48:44.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.545+0000 7f24574f5780 1 < completed write operation key=BV8C0VSJ1GTW tag=1AYPU5E0S5SS type=write size=987087 2023-12-03T11:48:44.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.546+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=IQUSTO63XPNP type=write size=821677 2023-12-03T11:48:44.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.548+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=O7WDAOJ8GTF6 type=del size=2581842 2023-12-03T11:48:44.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.549+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=41OSNVKS1IF4 type=del size=3933343 2023-12-03T11:48:44.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.551+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~LODZST3HBZRX.0 size=1048576 2023-12-03T11:48:44.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.552+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~LODZST3HBZRX.1 size=438588 2023-12-03T11:48:44.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.553+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=LODZST3HBZRX tag=YMC5OA1G1ZLJ type=write size=2535740 2023-12-03T11:48:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.555+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~GDWG8114K2VW.0 size=868671 2023-12-03T11:48:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.555+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=GDWG8114K2VW tag=EQHSZQBVQVN3 type=write size=1917247 2023-12-03T11:48:44.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.557+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=S10V5F8E5F53 type=del 2023-12-03T11:48:44.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.559+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~EKP0DABB93K1.0 size=1048576 2023-12-03T11:48:44.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.560+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~EKP0DABB93K1.1 size=359105 2023-12-03T11:48:44.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.560+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=EKP0DABB93K1 tag=9PEMYP21BVQU type=write size=2456257 2023-12-03T11:48:44.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.562+0000 7f24574f5780 1 < suggested remove operation key=WZMWCSAZX8RB tag=SGHKESXAIP0F type=del 2023-12-03T11:48:44.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.563+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=CBHSQXXJ8FW5 type=del size=3500421 2023-12-03T11:48:44.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.564+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.0 size=1048576 2023-12-03T11:48:44.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.565+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.1 size=1048576 2023-12-03T11:48:44.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.566+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LLLCPIIQ4WXH.2 size=630503 2023-12-03T11:48:44.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.568+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=LLLCPIIQ4WXH tag=AG9M5412JWPK type=write size=3776231 2023-12-03T11:48:44.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.569+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=R8FD2QX1S9NQ type=del size=1973093 2023-12-03T11:48:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.571+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=NKXVHG42JDG2 type=del 2023-12-03T11:48:44.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.572+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=GO7YDQB1GU3I type=del size=1713411 2023-12-03T11:48:44.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.573+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=FKVRVD3K7ONM type=del 2023-12-03T11:48:44.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.575+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=P5FPF2RD9Z46 type=del size=542017 2023-12-03T11:48:44.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.576+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=KYU91UK973EX type=del 2023-12-03T11:48:44.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.577+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=RYE04DMYNUFU type=del size=107171 2023-12-03T11:48:44.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.578+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=J3ZOBUO14X65 type=del 2023-12-03T11:48:44.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.580+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=SNJ32RU3XV0K type=write size=846980 2023-12-03T11:48:44.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.581+0000 7f24574f5780 1 < completed write operation key=863EOZDYRWH3 tag=BPUW6A8RIY0O type=write size=448089 2023-12-03T11:48:44.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.583+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.0 size=1048576 2023-12-03T11:48:44.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.584+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.1 size=1048576 2023-12-03T11:48:44.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.585+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.2 size=927286 2023-12-03T11:48:44.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.585+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=WOPLIMWBHIP2 tag=S9LHU05GB6G9 type=write size=4073014 2023-12-03T11:48:44.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.586+0000 7f24574f5780 1 < canceled operation key=SUBILSL92D96 tag=KNK1E7LZEQ4M type=write size=207888 2023-12-03T11:48:44.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.588+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=N13EEQF2KO3K type=del size=3878640 2023-12-03T11:48:44.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.589+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~B5BKK6I40RXD.0 size=231561 2023-12-03T11:48:44.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.591+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=B5BKK6I40RXD tag=UJNJ16ZS5WJ1 type=write size=1280137 2023-12-03T11:48:44.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.593+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.0 size=1048576 2023-12-03T11:48:44.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.594+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.1 size=1048576 2023-12-03T11:48:44.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.595+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.2 size=540543 2023-12-03T11:48:44.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.595+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=93A3EYRYNCGH tag=CV7OQYW1DX6F type=write size=3686271 2023-12-03T11:48:44.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.596+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=ZHGPIHHBRVZI type=del 2023-12-03T11:48:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.597+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=MKUUBR2M5W2X type=del size=1160013 2023-12-03T11:48:44.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.600+0000 7f24574f5780 1 < suggested remove operation key=XRHVYVADRWET tag=MHR0N5EOPJ1K type=del 2023-12-03T11:48:44.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.602+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~QN6KXXIMVSQC.0 size=988347 2023-12-03T11:48:44.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.602+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=QN6KXXIMVSQC tag=TETWFS34MUTR type=write size=2036923 2023-12-03T11:48:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.604+0000 7f24574f5780 1 < suggested remove operation key=BMKG19L35HLV tag=1SOQMITDDXS1 type=del 2023-12-03T11:48:44.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.606+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.0 size=1048576 2023-12-03T11:48:44.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.607+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.1 size=1048576 2023-12-03T11:48:44.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.608+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.2 size=869921 2023-12-03T11:48:44.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.608+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=ZT5PAMDH9S6T tag=YZYZQJ4CHU47 type=write size=4015649 2023-12-03T11:48:44.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.609+0000 7f24574f5780 1 < completed write operation key=U043YAPGUBT5 tag=IQUSTO63XPNP type=write size=821677 2023-12-03T11:48:44.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.610+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=U0JTNS3DFMPH type=del size=2002950 2023-12-03T11:48:44.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.612+0000 7f24574f5780 -1 < failed to complete operation key=BMKG19L35HLV tag=41OSNVKS1IF4 type=del size=3933343: (22) Invalid argument 2023-12-03T11:48:44.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.614+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~XLHCCPSQVXWW.0 size=862699 2023-12-03T11:48:44.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.614+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=XLHCCPSQVXWW tag=TGMHSXK82SQV type=write size=1911275 2023-12-03T11:48:44.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.615+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~GDWG8114K2VW.0 size=868671 2023-12-03T11:48:44.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.618+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=GDWG8114K2VW tag=EQHSZQBVQVN3 type=write size=1917247 2023-12-03T11:48:44.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.619+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=NAW1SDNK3SWJ type=del size=1007429 2023-12-03T11:48:44.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.621+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~EKP0DABB93K1.0 size=1048576 2023-12-03T11:48:44.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.622+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~EKP0DABB93K1.1 size=359105 2023-12-03T11:48:44.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.624+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=EKP0DABB93K1 tag=9PEMYP21BVQU type=write size=2456257 2023-12-03T11:48:44.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.625+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=C40YMWC9HQQO type=del size=2043683 2023-12-03T11:48:44.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.627+0000 7f24574f5780 1 < suggested remove operation key=XO42YZR60KQE tag=CBHSQXXJ8FW5 type=del 2023-12-03T11:48:44.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.630+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.0 size=1048576 2023-12-03T11:48:44.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.631+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.1 size=1048576 2023-12-03T11:48:44.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.631+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.2 size=967922 2023-12-03T11:48:44.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.631+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=GRDT27J3X2JR tag=I1WRL13ZAZ0G type=write size=4113650 2023-12-03T11:48:44.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.633+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=R8FD2QX1S9NQ type=del 2023-12-03T11:48:44.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.634+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=2NIGV44VX8ET type=del size=198825 2023-12-03T11:48:44.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.636+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=GO7YDQB1GU3I type=del size=1713411 2023-12-03T11:48:44.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.638+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.0 size=1048576 2023-12-03T11:48:44.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.639+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.1 size=1048576 2023-12-03T11:48:44.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.640+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.2 size=369496 2023-12-03T11:48:44.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.640+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=RQJ5U7BZ23PM tag=JLEHO4L381RO type=write size=3515224 2023-12-03T11:48:44.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.641+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=P5FPF2RD9Z46 type=del 2023-12-03T11:48:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.643+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~PGPVEYADVW0C.0 size=213785 2023-12-03T11:48:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.643+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=PGPVEYADVW0C tag=78Y22GQQCLBT type=write size=1262361 2023-12-03T11:48:44.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.645+0000 7f24574f5780 -1 < failed to complete operation key=XO42YZR60KQE tag=RYE04DMYNUFU type=del size=107171: (22) Invalid argument 2023-12-03T11:48:44.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.646+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=Y054MV3SYSO1 type=del size=123037 2023-12-03T11:48:44.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.647+0000 7f24574f5780 1 < canceled operation key=XZFHFS76WNR4 tag=SNJ32RU3XV0K type=write size=846980 2023-12-03T11:48:44.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.648+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=ERQ8W061DBZJ type=del size=1401053 2023-12-03T11:48:44.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.649+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.0 size=1048576 2023-12-03T11:48:44.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.649+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.1 size=1048576 2023-12-03T11:48:44.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.650+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~WOPLIMWBHIP2.2 size=927286 2023-12-03T11:48:44.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.652+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=WOPLIMWBHIP2 tag=S9LHU05GB6G9 type=write size=4073014 2023-12-03T11:48:44.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.655+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.0 size=1048576 2023-12-03T11:48:44.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.655+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.1 size=1048576 2023-12-03T11:48:44.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.656+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.2 size=91633 2023-12-03T11:48:44.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.656+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=0883F67FPURH tag=EHNQ7N4IQ2SE type=write size=3237361 2023-12-03T11:48:44.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.658+0000 7f24574f5780 -1 < failed to complete operation key=SUBILSL92D96 tag=N13EEQF2KO3K type=del size=3878640: (22) Invalid argument 2023-12-03T11:48:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.659+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=ZSDSLSR49KNQ type=del size=3789183 2023-12-03T11:48:44.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.660+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.0 size=1048576 2023-12-03T11:48:44.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.661+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.1 size=1048576 2023-12-03T11:48:44.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.662+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~93A3EYRYNCGH.2 size=540543 2023-12-03T11:48:44.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.664+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=93A3EYRYNCGH tag=CV7OQYW1DX6F type=write size=3686271 2023-12-03T11:48:44.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.666+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~9SNSMQGPPGH8.0 size=976813 2023-12-03T11:48:44.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.666+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=9SNSMQGPPGH8 tag=QXCX4KMJA2OG type=write size=2025389 2023-12-03T11:48:44.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.667+0000 7f24574f5780 1 < canceled operation key=DA6ZXCLKTC1L tag=MKUUBR2M5W2X type=del size=1160013 2023-12-03T11:48:44.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.669+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=AOUFBIMTH7LT type=del size=585290 2023-12-03T11:48:44.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.670+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~QN6KXXIMVSQC.0 size=988347 2023-12-03T11:48:44.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.672+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=QN6KXXIMVSQC tag=TETWFS34MUTR type=write size=2036923 2023-12-03T11:48:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.674+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IQP3VTQ0GSJP.0 size=911555 2023-12-03T11:48:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.674+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=IQP3VTQ0GSJP tag=Z822HVTA40HC type=write size=1960131 2023-12-03T11:48:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.675+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.0 size=1048576 2023-12-03T11:48:44.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.676+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.1 size=1048576 2023-12-03T11:48:44.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.677+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZT5PAMDH9S6T.2 size=869921 2023-12-03T11:48:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.679+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=ZT5PAMDH9S6T tag=YZYZQJ4CHU47 type=write size=4015649 2023-12-03T11:48:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.681+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~7YAYYLKTJ0WV.0 size=1048576 2023-12-03T11:48:44.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.682+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~7YAYYLKTJ0WV.1 size=187130 2023-12-03T11:48:44.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.682+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=7YAYYLKTJ0WV tag=NH44AMTSANXU type=write size=2284282 2023-12-03T11:48:44.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.683+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=U0JTNS3DFMPH type=del 2023-12-03T11:48:44.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.685+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=T1T74LX3MPNT type=del size=2406524 2023-12-03T11:48:44.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.686+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~XLHCCPSQVXWW.0 size=862699 2023-12-03T11:48:44.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.688+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=XLHCCPSQVXWW tag=TGMHSXK82SQV type=write size=1911275 2023-12-03T11:48:44.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.689+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=ZTDU7V5JYCMM type=del size=339486 2023-12-03T11:48:44.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.691+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=NAW1SDNK3SWJ type=del 2023-12-03T11:48:44.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.693+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~FIZODF5DPW35.0 size=1048576 2023-12-03T11:48:44.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.694+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~FIZODF5DPW35.1 size=636600 2023-12-03T11:48:44.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.694+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=FIZODF5DPW35 tag=HYO7HLNG4UTX type=write size=2733752 2023-12-03T11:48:44.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.695+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=C40YMWC9HQQO type=del size=2043683 2023-12-03T11:48:44.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.697+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~DU2SI62YPY3U.0 size=1048576 2023-12-03T11:48:44.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.698+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~DU2SI62YPY3U.1 size=742075 2023-12-03T11:48:44.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.698+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=DU2SI62YPY3U tag=GXUPBP9WWUD6 type=write size=2839227 2023-12-03T11:48:44.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.699+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.0 size=1048576 2023-12-03T11:48:44.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.700+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.1 size=1048576 2023-12-03T11:48:44.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.701+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~GRDT27J3X2JR.2 size=967922 2023-12-03T11:48:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.703+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=GRDT27J3X2JR tag=I1WRL13ZAZ0G type=write size=4113650 2023-12-03T11:48:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.704+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=7JO62Z9IF110 type=del size=1147006 2023-12-03T11:48:44.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.706+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=2NIGV44VX8ET type=del size=198825 2023-12-03T11:48:44.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.707+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=NUVXDMFYXXUH type=del size=1762851 2023-12-03T11:48:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.708+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.0 size=1048576 2023-12-03T11:48:44.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.709+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.1 size=1048576 2023-12-03T11:48:44.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.710+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~RQJ5U7BZ23PM.2 size=369496 2023-12-03T11:48:44.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.712+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=RQJ5U7BZ23PM tag=JLEHO4L381RO type=write size=3515224 2023-12-03T11:48:44.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.714+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=JJYRJ8NB2T82 type=del size=933990 2023-12-03T11:48:44.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.715+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~PGPVEYADVW0C.0 size=213785 2023-12-03T11:48:44.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.717+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=PGPVEYADVW0C tag=78Y22GQQCLBT type=write size=1262361 2023-12-03T11:48:44.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.718+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=C5DN5A46JLWZ type=write size=917832 2023-12-03T11:48:44.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.719+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=Y054MV3SYSO1 type=del 2023-12-03T11:48:44.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.721+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=ZJM3X6SRG3HA type=write size=775860 2023-12-03T11:48:44.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.722+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=ERQ8W061DBZJ type=del 2023-12-03T11:48:44.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.723+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=YBMG8YYZCBP8 type=del size=3101313 2023-12-03T11:48:44.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.724+0000 7f24574f5780 1 < canceled part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.0 size=1048576 2023-12-03T11:48:44.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.725+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.1 size=1048576 2023-12-03T11:48:44.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.726+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~0883F67FPURH.2 size=91633 2023-12-03T11:48:44.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.729+0000 7f24574f5780 1 < canceled multipart upload key=1QR0TE9DXU5Z upload=0883F67FPURH tag=EHNQ7N4IQ2SE type=write size=3237361 2023-12-03T11:48:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.731+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~52LOGE44T6ZP.0 size=418988 2023-12-03T11:48:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.731+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=52LOGE44T6ZP tag=L2X1VCLGHKTA type=write size=1467564 2023-12-03T11:48:44.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.733+0000 7f24574f5780 1 < suggested update operation key=BMKG19L35HLV tag=ZSDSLSR49KNQ type=del size=3789183 2023-12-03T11:48:44.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.735+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=3MJ0PMVYFVZI type=del size=4053486 2023-12-03T11:48:44.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.735+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~9SNSMQGPPGH8.0 size=976813 2023-12-03T11:48:44.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.737+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=9SNSMQGPPGH8 tag=QXCX4KMJA2OG type=write size=2025389 2023-12-03T11:48:44.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.739+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.0 size=1048576 2023-12-03T11:48:44.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.740+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.1 size=1048576 2023-12-03T11:48:44.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.740+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.2 size=17368 2023-12-03T11:48:44.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.740+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=LN4KVUNDFCY9 tag=F48M2RHT4AV3 type=write size=3163096 2023-12-03T11:48:44.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.742+0000 7f24574f5780 1 < canceled operation key=SC8UOVLQQ0VD tag=AOUFBIMTH7LT type=del size=585290 2023-12-03T11:48:44.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.743+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=4UJE8Q3PY868 type=del size=3030656 2023-12-03T11:48:44.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.744+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IQP3VTQ0GSJP.0 size=911555 2023-12-03T11:48:44.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.746+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=IQP3VTQ0GSJP tag=Z822HVTA40HC type=write size=1960131 2023-12-03T11:48:44.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.747+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=K5JR9ZNTYDOZ type=write size=278063 2023-12-03T11:48:44.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.748+0000 7f24574f5780 1 < canceled part key=_multipart_SUBILSL92D96.2~7YAYYLKTJ0WV.0 size=1048576 2023-12-03T11:48:44.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.748+0000 7f24574f5780 1 < canceled part key=_multipart_SUBILSL92D96.2~7YAYYLKTJ0WV.1 size=187130 2023-12-03T11:48:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.751+0000 7f24574f5780 1 < canceled multipart upload key=SUBILSL92D96 upload=7YAYYLKTJ0WV tag=NH44AMTSANXU type=write size=2284282 2023-12-03T11:48:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.752+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=2EKGVPBH1RKQ type=del size=1904603 2023-12-03T11:48:44.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.753+0000 7f24574f5780 1 < canceled operation key=1V1C59DP8YPF tag=T1T74LX3MPNT type=del size=2406524 2023-12-03T11:48:44.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.754+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=8T4M0GP6NDBN type=write size=560901 2023-12-03T11:48:44.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.756+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=ZTDU7V5JYCMM type=del size=339486 2023-12-03T11:48:44.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.758+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=29E5MHAUJEDN type=del size=3322053 2023-12-03T11:48:44.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.759+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~FIZODF5DPW35.0 size=1048576 2023-12-03T11:48:44.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.760+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~FIZODF5DPW35.1 size=636600 2023-12-03T11:48:44.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.762+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=FIZODF5DPW35 tag=HYO7HLNG4UTX type=write size=2733752 2023-12-03T11:48:44.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.764+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.0 size=1048576 2023-12-03T11:48:44.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.765+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.1 size=1048576 2023-12-03T11:48:44.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.765+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.2 size=934132 2023-12-03T11:48:44.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.765+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=88VO0AY6ST5N tag=DC2K0AP8NLXZ type=write size=4079860 2023-12-03T11:48:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.767+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~DU2SI62YPY3U.0 size=1048576 2023-12-03T11:48:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.768+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~DU2SI62YPY3U.1 size=742075 2023-12-03T11:48:44.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.771+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=DU2SI62YPY3U tag=GXUPBP9WWUD6 type=write size=2839227 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.773+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~MKBHWUAH7G12.0 size=1048576 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.774+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~MKBHWUAH7G12.1 size=527154 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.774+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=MKBHWUAH7G12 tag=YUMI5K57GE2R type=write size=2624306 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.776+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=7JO62Z9IF110 type=del 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.777+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=1SFIB3XT0UMF type=del size=2728044 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.779+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=NUVXDMFYXXUH type=del size=1762851 2023-12-03T11:48:44.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.780+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=A2M0EQWKLE81 type=del size=1645042 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.781+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=JJYRJ8NB2T82 type=del 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.782+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=GJ9UY7476CWC type=del size=2258675 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.783+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=C5DN5A46JLWZ type=write size=917832 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.784+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=R666CCPTB4J5 type=del size=1971135 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.787+0000 7f24574f5780 1 < suggested update operation key=BMKG19L35HLV tag=ZJM3X6SRG3HA type=write size=775860 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.788+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=FOGGXCQDRNF9 type=del size=1848200 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.789+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=YBMG8YYZCBP8 type=del 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.791+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.0 size=1048576 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.792+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.1 size=1048576 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.793+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.2 size=340782 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.793+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=O0PRVN27ZMS3 tag=LN54I1HCXM38 type=write size=3486510 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.794+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~52LOGE44T6ZP.0 size=418988 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.796+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=52LOGE44T6ZP tag=L2X1VCLGHKTA type=write size=1467564 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.798+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2MGS7KX9BZYN.0 size=332520 2023-12-03T11:48:44.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.798+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=2MGS7KX9BZYN tag=PIVWSSA0ZRXW type=write size=1381096 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.799+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=3MJ0PMVYFVZI type=del 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.801+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=TEAMRKBKLVFD type=del size=2265273 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.802+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.0 size=1048576 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.803+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.1 size=1048576 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.804+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~LN4KVUNDFCY9.2 size=17368 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.806+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=LN4KVUNDFCY9 tag=F48M2RHT4AV3 type=write size=3163096 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.807+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=KJAY846Z25V6 type=del size=2958270 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.808+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=4UJE8Q3PY868 type=del 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.809+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=D6DN7H8TEJRG type=del size=970529 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.811+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=K5JR9ZNTYDOZ type=write size=278063 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.813+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~0J0XLBW099VR.0 size=490490 2023-12-03T11:48:44.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.813+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=0J0XLBW099VR tag=YY3M7TXFQ0QN type=write size=1539066 2023-12-03T11:48:44.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.815+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=2EKGVPBH1RKQ type=del size=1904603 2023-12-03T11:48:44.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.817+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.0 size=1048576 2023-12-03T11:48:44.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.818+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.1 size=1048576 2023-12-03T11:48:44.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.819+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.2 size=719168 2023-12-03T11:48:44.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.819+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=DOF7HY10Q48W tag=BAZO5VYCCNU1 type=write size=3864896 2023-12-03T11:48:44.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.820+0000 7f24574f5780 1 < completed write operation key=S0EH0Q0UA8HG tag=8T4M0GP6NDBN type=write size=560901 2023-12-03T11:48:44.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.822+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=F9U0UYY55YZI type=del size=811 2023-12-03T11:48:44.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.824+0000 7f24574f5780 1 < suggested remove operation key=XO42YZR60KQE tag=29E5MHAUJEDN type=del 2023-12-03T11:48:44.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.825+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=JNHCN8H9EHIR type=write size=338113 2023-12-03T11:48:44.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.827+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.0 size=1048576 2023-12-03T11:48:44.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.827+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.1 size=1048576 2023-12-03T11:48:44.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.828+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~88VO0AY6ST5N.2 size=934132 2023-12-03T11:48:44.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.830+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=88VO0AY6ST5N tag=DC2K0AP8NLXZ type=write size=4079860 2023-12-03T11:48:44.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.832+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.0 size=1048576 2023-12-03T11:48:44.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.834+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.1 size=1048576 2023-12-03T11:48:44.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.835+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.2 size=171085 2023-12-03T11:48:44.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.835+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=92NXY2JHZKA7 tag=T5UFQ2NYZTJ9 type=write size=3316813 2023-12-03T11:48:44.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.836+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~MKBHWUAH7G12.0 size=1048576 2023-12-03T11:48:44.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.836+0000 7f24574f5780 1 < canceled part key=_multipart_LS4OEAGZWQXC.2~MKBHWUAH7G12.1 size=527154 2023-12-03T11:48:44.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.838+0000 7f24574f5780 1 < canceled multipart upload key=LS4OEAGZWQXC upload=MKBHWUAH7G12 tag=YUMI5K57GE2R type=write size=2624306 2023-12-03T11:48:44.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.839+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=QDQRW77LGHFO type=del size=3058193 2023-12-03T11:48:44.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.842+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=1SFIB3XT0UMF type=del 2023-12-03T11:48:44.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.843+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=G9QKYXW878SS type=write size=154580 2023-12-03T11:48:44.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.846+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=A2M0EQWKLE81 type=del 2023-12-03T11:48:44.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.848+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~YKDRY4EKPZAH.0 size=259761 2023-12-03T11:48:44.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.848+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=YKDRY4EKPZAH tag=7TMQGWXB0ERG type=write size=1308337 2023-12-03T11:48:44.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.849+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=GJ9UY7476CWC type=del 2023-12-03T11:48:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.851+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~V79K2ATW7GBQ.0 size=290016 2023-12-03T11:48:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.851+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=V79K2ATW7GBQ tag=4VB9UE50KZU4 type=write size=1338592 2023-12-03T11:48:44.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.853+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=R666CCPTB4J5 type=del 2023-12-03T11:48:44.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.854+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=6K9IHMZ3TWI0 type=del size=2679124 2023-12-03T11:48:44.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.855+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=FOGGXCQDRNF9 type=del 2023-12-03T11:48:44.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.857+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.0 size=1048576 2023-12-03T11:48:44.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.858+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.1 size=1048576 2023-12-03T11:48:44.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.859+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.2 size=368113 2023-12-03T11:48:44.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.859+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=17T8CJFGO1SG tag=P6B02MV6B9I6 type=write size=3513841 2023-12-03T11:48:44.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.861+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.0 size=1048576 2023-12-03T11:48:44.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.861+0000 7f24574f5780 1 < canceled part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.1 size=1048576 2023-12-03T11:48:44.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.861+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~O0PRVN27ZMS3.2 size=340782 2023-12-03T11:48:44.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.863+0000 7f24574f5780 1 < canceled multipart upload key=1QR0TE9DXU5Z upload=O0PRVN27ZMS3 tag=LN54I1HCXM38 type=write size=3486510 2023-12-03T11:48:44.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.865+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=131M8O2J0OY3 type=del size=983360 2023-12-03T11:48:44.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.866+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2MGS7KX9BZYN.0 size=332520 2023-12-03T11:48:44.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.868+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=2MGS7KX9BZYN tag=PIVWSSA0ZRXW type=write size=1381096 2023-12-03T11:48:44.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.870+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.0 size=1048576 2023-12-03T11:48:44.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.871+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.1 size=1048576 2023-12-03T11:48:44.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.872+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.2 size=776779 2023-12-03T11:48:44.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.872+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=DLUMV7SVURCI tag=BD6QZEI4CZ5W type=write size=3922507 2023-12-03T11:48:44.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.874+0000 7f24574f5780 1 < suggested update operation key=MY3K9M4QRS3L tag=TEAMRKBKLVFD type=del size=2265273 2023-12-03T11:48:44.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.876+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~442BMN42GHC6.0 size=1048576 2023-12-03T11:48:44.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.877+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~442BMN42GHC6.1 size=245962 2023-12-03T11:48:44.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.877+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=442BMN42GHC6 tag=2MYQA1UPTTTF type=write size=2343114 2023-12-03T11:48:44.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.878+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=KJAY846Z25V6 type=del 2023-12-03T11:48:44.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.879+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=C82FXCJ2GEIS type=write size=191961 2023-12-03T11:48:44.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.881+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=D6DN7H8TEJRG type=del 2023-12-03T11:48:44.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.882+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=4SGEYW8L66SN type=del size=1091153 2023-12-03T11:48:44.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.883+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~0J0XLBW099VR.0 size=490490 2023-12-03T11:48:44.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.885+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=0J0XLBW099VR tag=YY3M7TXFQ0QN type=write size=1539066 2023-12-03T11:48:44.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.887+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~MK88V6AVE9ZD.0 size=1048576 2023-12-03T11:48:44.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.888+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~MK88V6AVE9ZD.1 size=167709 2023-12-03T11:48:44.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.888+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=MK88V6AVE9ZD tag=3PAIPFDL2RO7 type=write size=2264861 2023-12-03T11:48:44.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.889+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.0 size=1048576 2023-12-03T11:48:44.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.890+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.1 size=1048576 2023-12-03T11:48:44.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.891+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DOF7HY10Q48W.2 size=719168 2023-12-03T11:48:44.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.893+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=DOF7HY10Q48W tag=BAZO5VYCCNU1 type=write size=3864896 2023-12-03T11:48:44.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.894+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=LUS8M5FZN02U type=del size=2270140 2023-12-03T11:48:44.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.895+0000 7f24574f5780 -1 < failed to complete operation key=QVLPCWC4T8N8 tag=F9U0UYY55YZI type=del size=811: (22) Invalid argument 2023-12-03T11:48:44.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.896+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=E81DB4HAPWHQ type=del size=3965974 2023-12-03T11:48:44.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.898+0000 7f24574f5780 1 < canceled operation key=S7Q5TCM499H1 tag=JNHCN8H9EHIR type=write size=338113 2023-12-03T11:48:44.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.899+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=PC0IV2SM8F1C type=write size=450630 2023-12-03T11:48:44.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.900+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.0 size=1048576 2023-12-03T11:48:44.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.901+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.1 size=1048576 2023-12-03T11:48:44.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.902+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~92NXY2JHZKA7.2 size=171085 2023-12-03T11:48:44.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.904+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=92NXY2JHZKA7 tag=T5UFQ2NYZTJ9 type=write size=3316813 2023-12-03T11:48:44.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.905+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=TDMESNEEA3T4 type=del size=357033 2023-12-03T11:48:44.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.906+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=QDQRW77LGHFO type=del 2023-12-03T11:48:44.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.907+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=258AY5LBS35P type=del size=3533092 2023-12-03T11:48:44.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.908+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=G9QKYXW878SS type=write size=154580 2023-12-03T11:48:44.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.910+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~G112FUP17VTK.0 size=647572 2023-12-03T11:48:44.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.910+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=G112FUP17VTK tag=7KZT2CDDXRP3 type=write size=1696148 2023-12-03T11:48:44.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.911+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~YKDRY4EKPZAH.0 size=259761 2023-12-03T11:48:44.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.913+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=YKDRY4EKPZAH tag=7TMQGWXB0ERG type=write size=1308337 2023-12-03T11:48:44.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.914+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=1XLD06NXKIGI type=del size=3400414 2023-12-03T11:48:44.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.916+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~V79K2ATW7GBQ.0 size=290016 2023-12-03T11:48:44.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.917+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=V79K2ATW7GBQ tag=4VB9UE50KZU4 type=write size=1338592 2023-12-03T11:48:44.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.920+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.0 size=1048576 2023-12-03T11:48:44.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.920+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.1 size=1048576 2023-12-03T11:48:44.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.921+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.2 size=75888 2023-12-03T11:48:44.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.921+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=T70JWME592O6 tag=S8JLWHCO0RCG type=write size=3221616 2023-12-03T11:48:44.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.924+0000 7f24574f5780 1 < suggested update operation key=RABUR02D2ZMD tag=6K9IHMZ3TWI0 type=del size=2679124 2023-12-03T11:48:44.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.925+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=KRH200HWWIGL type=del size=3104169 2023-12-03T11:48:44.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.926+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.0 size=1048576 2023-12-03T11:48:44.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.927+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.1 size=1048576 2023-12-03T11:48:44.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.928+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~17T8CJFGO1SG.2 size=368113 2023-12-03T11:48:44.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.930+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=17T8CJFGO1SG tag=P6B02MV6B9I6 type=write size=3513841 2023-12-03T11:48:44.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.932+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~6345YFKTZXAF.0 size=100609 2023-12-03T11:48:44.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.932+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=6345YFKTZXAF tag=VCAH3KPJVB25 type=write size=1149185 2023-12-03T11:48:44.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.933+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=131M8O2J0OY3 type=del 2023-12-03T11:48:44.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.935+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~I5SZZF65RZ69.0 size=296083 2023-12-03T11:48:44.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.935+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=I5SZZF65RZ69 tag=8GV19SKE0C4G type=write size=1344659 2023-12-03T11:48:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.936+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.0 size=1048576 2023-12-03T11:48:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.937+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.1 size=1048576 2023-12-03T11:48:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.938+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~DLUMV7SVURCI.2 size=776779 2023-12-03T11:48:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.940+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=DLUMV7SVURCI tag=BD6QZEI4CZ5W type=write size=3922507 2023-12-03T11:48:44.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.942+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~7LNS4SX9WB85.0 size=659547 2023-12-03T11:48:44.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.942+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=7LNS4SX9WB85 tag=CWKZ07F1VJBT type=write size=1708123 2023-12-03T11:48:44.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.944+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~442BMN42GHC6.0 size=1048576 2023-12-03T11:48:44.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.945+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~442BMN42GHC6.1 size=245962 2023-12-03T11:48:44.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.947+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=442BMN42GHC6 tag=2MYQA1UPTTTF type=write size=2343114 2023-12-03T11:48:44.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.948+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=GI3MZT1LX1AQ type=del size=2827941 2023-12-03T11:48:44.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.950+0000 7f24574f5780 1 < completed write operation key=S0EH0Q0UA8HG tag=C82FXCJ2GEIS type=write size=191961 2023-12-03T11:48:44.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.951+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=CURBVL9RFU5D type=del size=2432279 2023-12-03T11:48:44.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.953+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=4SGEYW8L66SN type=del 2023-12-03T11:48:44.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.954+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=BMPABOA0KJM8 type=del size=376853 2023-12-03T11:48:44.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.955+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~MK88V6AVE9ZD.0 size=1048576 2023-12-03T11:48:44.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.956+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~MK88V6AVE9ZD.1 size=167709 2023-12-03T11:48:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.958+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=MK88V6AVE9ZD tag=3PAIPFDL2RO7 type=write size=2264861 2023-12-03T11:48:44.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.960+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.0 size=1048576 2023-12-03T11:48:44.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.961+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.1 size=1048576 2023-12-03T11:48:44.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.962+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.2 size=826619 2023-12-03T11:48:44.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.962+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=DVZAL8K8GFU9 tag=1KI90NMIKT5I type=write size=3972347 2023-12-03T11:48:44.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.963+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=LUS8M5FZN02U type=del 2023-12-03T11:48:44.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.964+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=V6QRQWTIZYDJ type=del size=3027857 2023-12-03T11:48:44.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.966+0000 7f24574f5780 1 < suggested remove operation key=WZMWCSAZX8RB tag=E81DB4HAPWHQ type=del 2023-12-03T11:48:44.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.968+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.0 size=1048576 2023-12-03T11:48:44.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.969+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.1 size=1048576 2023-12-03T11:48:44.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.970+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.2 size=619957 2023-12-03T11:48:44.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.970+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=2DOK8OYWY87B tag=DTHFXQEUX8OL type=write size=3765685 2023-12-03T11:48:44.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.971+0000 7f24574f5780 1 < canceled operation key=MY3K9M4QRS3L tag=PC0IV2SM8F1C type=write size=450630 2023-12-03T11:48:44.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.973+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=2NFXDWMI4IEI type=del size=3348359 2023-12-03T11:48:44.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.974+0000 7f24574f5780 1 < canceled operation key=WD7U31FW3NGZ tag=TDMESNEEA3T4 type=del size=357033 2023-12-03T11:48:44.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.975+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=QNMUA8N65H9J type=del size=570611 2023-12-03T11:48:44.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.976+0000 7f24574f5780 1 < canceled operation key=XO42YZR60KQE tag=258AY5LBS35P type=del size=3533092 2023-12-03T11:48:44.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.977+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=AD36GOP5I8SK type=del size=2462241 2023-12-03T11:48:44.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.979+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~G112FUP17VTK.0 size=647572 2023-12-03T11:48:44.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.981+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=G112FUP17VTK tag=7KZT2CDDXRP3 type=write size=1696148 2023-12-03T11:48:44.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.982+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=W7204UQ53Q4J type=write size=16183 2023-12-03T11:48:44.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.983+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=1XLD06NXKIGI type=del 2023-12-03T11:48:44.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.984+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=FU01ES4W5AQW type=del size=3135066 2023-12-03T11:48:44.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.985+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.0 size=1048576 2023-12-03T11:48:44.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.986+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.1 size=1048576 2023-12-03T11:48:44.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.988+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~T70JWME592O6.2 size=75888 2023-12-03T11:48:44.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.990+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=T70JWME592O6 tag=S8JLWHCO0RCG type=write size=3221616 2023-12-03T11:48:44.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.991+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=QTQYHUACF5T3 type=write size=811500 2023-12-03T11:48:44.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.992+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=KRH200HWWIGL type=del 2023-12-03T11:48:44.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.994+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.0 size=1048576 2023-12-03T11:48:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.995+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.1 size=1048576 2023-12-03T11:48:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.996+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.2 size=205771 2023-12-03T11:48:44.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.996+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=6RWBYM769T94 tag=MWWJQS3VUJ87 type=write size=3351499 2023-12-03T11:48:44.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:44.998+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~6345YFKTZXAF.0 size=100609 2023-12-03T11:48:45.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.000+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=6345YFKTZXAF tag=VCAH3KPJVB25 type=write size=1149185 2023-12-03T11:48:45.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.002+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~BAI3RP5VCUT8.0 size=1048576 2023-12-03T11:48:45.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.003+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~BAI3RP5VCUT8.1 size=597770 2023-12-03T11:48:45.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.003+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=BAI3RP5VCUT8 tag=1J1NA696LCOP type=write size=2694922 2023-12-03T11:48:45.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.004+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~I5SZZF65RZ69.0 size=296083 2023-12-03T11:48:45.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.007+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=I5SZZF65RZ69 tag=8GV19SKE0C4G type=write size=1344659 2023-12-03T11:48:45.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.008+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=OARFR96JT3RH type=write size=239379 2023-12-03T11:48:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.009+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~7LNS4SX9WB85.0 size=659547 2023-12-03T11:48:45.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.011+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=7LNS4SX9WB85 tag=CWKZ07F1VJBT type=write size=1708123 2023-12-03T11:48:45.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.013+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~6LD9FRG4QWAV.0 size=1048576 2023-12-03T11:48:45.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.014+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~6LD9FRG4QWAV.1 size=1044089 2023-12-03T11:48:45.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.014+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=6LD9FRG4QWAV tag=FYQ2B1HLZC6F type=write size=3141241 2023-12-03T11:48:45.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.015+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=GI3MZT1LX1AQ type=del 2023-12-03T11:48:45.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.016+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=98FAYYPJ58UD type=write size=137111 2023-12-03T11:48:45.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.018+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=CURBVL9RFU5D type=del 2023-12-03T11:48:45.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.020+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~MHGCFB0NUD3Q.0 size=307466 2023-12-03T11:48:45.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.020+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=MHGCFB0NUD3Q tag=C9RHSR3XI6PI type=write size=1356042 2023-12-03T11:48:45.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.020+0000 7f24574f5780 -1 < no bi entry to suggest for operation key=WZMWCSAZX8RB tag=BMPABOA0KJM8 type=del size=376853: (2) No such file or directory 2023-12-03T11:48:45.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.022+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.0 size=1048576 2023-12-03T11:48:45.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.023+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.1 size=1048576 2023-12-03T11:48:45.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.024+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.2 size=736372 2023-12-03T11:48:45.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.024+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=IZJ6CNK8G97N tag=7QR9DUJW6OX2 type=write size=3882100 2023-12-03T11:48:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.025+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.0 size=1048576 2023-12-03T11:48:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.025+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.1 size=1048576 2023-12-03T11:48:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.026+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~DVZAL8K8GFU9.2 size=826619 2023-12-03T11:48:45.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.028+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=DVZAL8K8GFU9 tag=1KI90NMIKT5I type=write size=3972347 2023-12-03T11:48:45.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.029+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=BUWUXNSJJ04J type=del size=2031484 2023-12-03T11:48:45.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.031+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=V6QRQWTIZYDJ type=del 2023-12-03T11:48:45.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.032+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=4N5IOVFAVWF6 type=write size=778433 2023-12-03T11:48:45.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.033+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.0 size=1048576 2023-12-03T11:48:45.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.034+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.1 size=1048576 2023-12-03T11:48:45.036 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.035+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2DOK8OYWY87B.2 size=619957 2023-12-03T11:48:45.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.037+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=2DOK8OYWY87B tag=DTHFXQEUX8OL type=write size=3765685 2023-12-03T11:48:45.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.039+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~BNX8CNBOME32.0 size=1036959 2023-12-03T11:48:45.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.039+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=BNX8CNBOME32 tag=4CNI1O74B9K5 type=write size=2085535 2023-12-03T11:48:45.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.040+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=2NFXDWMI4IEI type=del 2023-12-03T11:48:45.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.042+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=0NVH1YW6EPRN type=del size=108019 2023-12-03T11:48:45.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.043+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=QNMUA8N65H9J type=del 2023-12-03T11:48:45.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.044+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=ACPX9GOL69EH type=del size=2536731 2023-12-03T11:48:45.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.045+0000 7f24574f5780 1 < canceled operation key=DA6ZXCLKTC1L tag=AD36GOP5I8SK type=del size=2462241 2023-12-03T11:48:45.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.046+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=B2STRF90YGAC type=write size=672682 2023-12-03T11:48:45.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.049+0000 7f24574f5780 1 < suggested remove operation key=XZFHFS76WNR4 tag=W7204UQ53Q4J type=write 2023-12-03T11:48:45.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.051+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.0 size=1048576 2023-12-03T11:48:45.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.052+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.1 size=1048576 2023-12-03T11:48:45.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.053+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.2 size=784758 2023-12-03T11:48:45.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.053+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=952TUQW348VN tag=6MKOHKQAYCI4 type=write size=3930486 2023-12-03T11:48:45.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.054+0000 7f24574f5780 1 < canceled operation key=XO42YZR60KQE tag=FU01ES4W5AQW type=del size=3135066 2023-12-03T11:48:45.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.055+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=3NHS6JXKJ1C4 type=del size=1815423 2023-12-03T11:48:45.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.056+0000 7f24574f5780 1 < completed write operation key=1V1C59DP8YPF tag=QTQYHUACF5T3 type=write size=811500 2023-12-03T11:48:45.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.057+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=U55ZSFNOAVC7 type=del size=739369 2023-12-03T11:48:45.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.059+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.0 size=1048576 2023-12-03T11:48:45.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.060+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.1 size=1048576 2023-12-03T11:48:45.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.062+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~6RWBYM769T94.2 size=205771 2023-12-03T11:48:45.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.068+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=6RWBYM769T94 tag=MWWJQS3VUJ87 type=write size=3351499 2023-12-03T11:48:45.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.069+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=S1Q675P5N40F type=del size=2844168 2023-12-03T11:48:45.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.070+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~BAI3RP5VCUT8.0 size=1048576 2023-12-03T11:48:45.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.070+0000 7f24574f5780 1 < canceled part key=_multipart_PS1OYWI44L80.2~BAI3RP5VCUT8.1 size=597770 2023-12-03T11:48:45.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.072+0000 7f24574f5780 1 < canceled multipart upload key=PS1OYWI44L80 upload=BAI3RP5VCUT8 tag=1J1NA696LCOP type=write size=2694922 2023-12-03T11:48:45.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.073+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=2EASOLJ36IHR type=write size=449894 2023-12-03T11:48:45.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.075+0000 7f24574f5780 1 < suggested remove operation key=PS1OYWI44L80 tag=OARFR96JT3RH type=write 2023-12-03T11:48:45.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.077+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=K8X42L7Y0Q22 type=del size=683513 2023-12-03T11:48:45.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.078+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~6LD9FRG4QWAV.0 size=1048576 2023-12-03T11:48:45.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.078+0000 7f24574f5780 1 < canceled part key=_multipart_WD7U31FW3NGZ.2~6LD9FRG4QWAV.1 size=1044089 2023-12-03T11:48:45.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.080+0000 7f24574f5780 1 < canceled multipart upload key=WD7U31FW3NGZ upload=6LD9FRG4QWAV tag=FYQ2B1HLZC6F type=write size=3141241 2023-12-03T11:48:45.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.081+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=33SQ549N90SE type=del size=3784283 2023-12-03T11:48:45.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.082+0000 7f24574f5780 1 < completed write operation key=6WY91DFIRUN8 tag=98FAYYPJ58UD type=write size=137111 2023-12-03T11:48:45.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.083+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=3QX9B3IMQ6DY type=write size=578874 2023-12-03T11:48:45.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.083+0000 7f24574f5780 1 < canceled part key=_multipart_RABUR02D2ZMD.2~MHGCFB0NUD3Q.0 size=307466 2023-12-03T11:48:45.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.085+0000 7f24574f5780 1 < canceled multipart upload key=RABUR02D2ZMD upload=MHGCFB0NUD3Q tag=C9RHSR3XI6PI type=write size=1356042 2023-12-03T11:48:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.086+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=7W0T82KHPA3P type=write size=414970 2023-12-03T11:48:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.087+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.0 size=1048576 2023-12-03T11:48:45.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.088+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.1 size=1048576 2023-12-03T11:48:45.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.089+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~IZJ6CNK8G97N.2 size=736372 2023-12-03T11:48:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.091+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=IZJ6CNK8G97N tag=7QR9DUJW6OX2 type=write size=3882100 2023-12-03T11:48:45.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.093+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=9QUVV4KZLZ47 type=write size=560187 2023-12-03T11:48:45.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.094+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=BUWUXNSJJ04J type=del 2023-12-03T11:48:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.095+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=XUXW0Q137J6Q type=del size=297415 2023-12-03T11:48:45.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.096+0000 7f24574f5780 1 < completed write operation key=RABUR02D2ZMD tag=4N5IOVFAVWF6 type=write size=778433 2023-12-03T11:48:45.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.099+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~VFCK5HEKE66A.0 size=1048576 2023-12-03T11:48:45.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.100+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~VFCK5HEKE66A.1 size=940381 2023-12-03T11:48:45.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.100+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=VFCK5HEKE66A tag=PQ0IITJVUQHK type=write size=3037533 2023-12-03T11:48:45.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.101+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~BNX8CNBOME32.0 size=1036959 2023-12-03T11:48:45.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.103+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=BNX8CNBOME32 tag=4CNI1O74B9K5 type=write size=2085535 2023-12-03T11:48:45.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.104+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=XDWTVFD5HPRO type=write size=341494 2023-12-03T11:48:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.105+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=0NVH1YW6EPRN type=del 2023-12-03T11:48:45.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.106+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=TQO6GI5KLD1M type=del size=795081 2023-12-03T11:48:45.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.108+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=ACPX9GOL69EH type=del 2023-12-03T11:48:45.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.110+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~GKXFAB4YHTHA.0 size=1048576 2023-12-03T11:48:45.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.111+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~GKXFAB4YHTHA.1 size=981098 2023-12-03T11:48:45.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.111+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=GKXFAB4YHTHA tag=OF6NH2BXT9GZ type=write size=3078250 2023-12-03T11:48:45.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.113+0000 7f24574f5780 1 < completed write operation key=U043YAPGUBT5 tag=B2STRF90YGAC type=write size=672682 2023-12-03T11:48:45.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.114+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=N6SHLESTZ1Q8 type=del size=1163600 2023-12-03T11:48:45.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.116+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.0 size=1048576 2023-12-03T11:48:45.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.116+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.1 size=1048576 2023-12-03T11:48:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.117+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~952TUQW348VN.2 size=784758 2023-12-03T11:48:45.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.119+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=952TUQW348VN tag=6MKOHKQAYCI4 type=write size=3930486 2023-12-03T11:48:45.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.120+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=68YPE0YIQLM3 type=del size=3475678 2023-12-03T11:48:45.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.122+0000 7f24574f5780 1 < suggested update operation key=WD7U31FW3NGZ tag=3NHS6JXKJ1C4 type=del size=1815423 2023-12-03T11:48:45.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.123+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=9BLCY5RGGXFY type=del size=3845289 2023-12-03T11:48:45.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.124+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=U55ZSFNOAVC7 type=del 2023-12-03T11:48:45.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.127+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.0 size=1048576 2023-12-03T11:48:45.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.128+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.1 size=1048576 2023-12-03T11:48:45.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.128+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.2 size=257935 2023-12-03T11:48:45.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.128+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=GCHDWC3DWMSB tag=APNGL7K40ZSN type=write size=3403663 2023-12-03T11:48:45.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.131+0000 7f24574f5780 1 < suggested update operation key=9LKFY1U0JRBQ tag=S1Q675P5N40F type=del size=2844168 2023-12-03T11:48:45.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.132+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=31MMTI9OG864 type=del size=165056 2023-12-03T11:48:45.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.135+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=2EASOLJ36IHR type=write size=449894 2023-12-03T11:48:45.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.136+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=2BXFG17RUBNT type=del size=2130368 2023-12-03T11:48:45.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.137+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=K8X42L7Y0Q22 type=del 2023-12-03T11:48:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.138+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=EQYFEDICM9VW type=del size=2011768 2023-12-03T11:48:45.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.139+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=33SQ549N90SE type=del 2023-12-03T11:48:45.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.140+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=ODCF77TE9QHQ type=write size=228122 2023-12-03T11:48:45.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.142+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=3QX9B3IMQ6DY type=write size=578874 2023-12-03T11:48:45.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.144+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.0 size=1048576 2023-12-03T11:48:45.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.145+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.1 size=1048576 2023-12-03T11:48:45.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.146+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.2 size=575396 2023-12-03T11:48:45.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.146+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=WVOBOKSUTG11 tag=3QYG0QVAR7WB type=write size=3721124 2023-12-03T11:48:45.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.147+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=7W0T82KHPA3P type=write size=414970 2023-12-03T11:48:45.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.149+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=T42UB33MFDP0 type=del size=206284 2023-12-03T11:48:45.151 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.150+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=9QUVV4KZLZ47 type=write size=560187 2023-12-03T11:48:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.152+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.0 size=1048576 2023-12-03T11:48:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.153+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.1 size=1048576 2023-12-03T11:48:45.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.154+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.2 size=533895 2023-12-03T11:48:45.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.154+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=F26ELAS2DOGX tag=GQE9B7L4SWQW type=write size=3679623 2023-12-03T11:48:45.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.155+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=XUXW0Q137J6Q type=del 2023-12-03T11:48:45.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.156+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=KVNQ4N1SQQOS type=del size=3360997 2023-12-03T11:48:45.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.158+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~VFCK5HEKE66A.0 size=1048576 2023-12-03T11:48:45.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.159+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~VFCK5HEKE66A.1 size=940381 2023-12-03T11:48:45.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.161+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=VFCK5HEKE66A tag=PQ0IITJVUQHK type=write size=3037533 2023-12-03T11:48:45.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.163+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~2CD24K3OQIUD.0 size=1048576 2023-12-03T11:48:45.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.164+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~2CD24K3OQIUD.1 size=157813 2023-12-03T11:48:45.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.164+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=2CD24K3OQIUD tag=N5ZILBWRX1NA type=write size=2254965 2023-12-03T11:48:45.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.167+0000 7f24574f5780 1 < suggested remove operation key=SUBILSL92D96 tag=XDWTVFD5HPRO type=write 2023-12-03T11:48:45.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.168+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=3DEQ7OB14S4N type=del size=816941 2023-12-03T11:48:45.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.170+0000 7f24574f5780 1 < suggested update operation key=PGTGVLK4DBN0 tag=TQO6GI5KLD1M type=del size=795081 2023-12-03T11:48:45.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.173+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~HOGX3O7CZW7D.0 size=1048576 2023-12-03T11:48:45.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.174+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~HOGX3O7CZW7D.1 size=778161 2023-12-03T11:48:45.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.174+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=HOGX3O7CZW7D tag=G3O554CDJ32K type=write size=2875313 2023-12-03T11:48:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.175+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~GKXFAB4YHTHA.0 size=1048576 2023-12-03T11:48:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.176+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~GKXFAB4YHTHA.1 size=981098 2023-12-03T11:48:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.178+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=GKXFAB4YHTHA tag=OF6NH2BXT9GZ type=write size=3078250 2023-12-03T11:48:45.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.180+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=3PD8W1060O55 type=del size=795408 2023-12-03T11:48:45.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.181+0000 7f24574f5780 1 < canceled operation key=3EOEF9UMATIE tag=N6SHLESTZ1Q8 type=del size=1163600 2023-12-03T11:48:45.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.182+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=CQXU1HJ2A76L type=del size=3776437 2023-12-03T11:48:45.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.184+0000 7f24574f5780 -1 < failed to complete operation key=WD7U31FW3NGZ tag=68YPE0YIQLM3 type=del size=3475678: (22) Invalid argument 2023-12-03T11:48:45.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.185+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=SNQ0VFFPWKP6 type=del size=3219121 2023-12-03T11:48:45.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.187+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=9BLCY5RGGXFY type=del size=3845289 2023-12-03T11:48:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.189+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~IFJCELMFWWSY.0 size=853330 2023-12-03T11:48:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.189+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=IFJCELMFWWSY tag=7D0VJAW18EQ1 type=write size=1901906 2023-12-03T11:48:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.190+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.0 size=1048576 2023-12-03T11:48:45.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.191+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.1 size=1048576 2023-12-03T11:48:45.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.192+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~GCHDWC3DWMSB.2 size=257935 2023-12-03T11:48:45.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.194+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=GCHDWC3DWMSB tag=APNGL7K40ZSN type=write size=3403663 2023-12-03T11:48:45.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.195+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=V2TRJOJCGTIW type=del size=3252044 2023-12-03T11:48:45.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.196+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=31MMTI9OG864 type=del 2023-12-03T11:48:45.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.197+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=HO79DZ811M8K type=write size=853732 2023-12-03T11:48:45.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.198+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=2BXFG17RUBNT type=del 2023-12-03T11:48:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.200+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=1S4U6ZJDOD1T type=write size=255138 2023-12-03T11:48:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.201+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=EQYFEDICM9VW type=del 2023-12-03T11:48:45.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.202+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=E25PRFFD0CBE type=del size=742499 2023-12-03T11:48:45.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.203+0000 7f24574f5780 1 < completed write operation key=3EOEF9UMATIE tag=ODCF77TE9QHQ type=write size=228122 2023-12-03T11:48:45.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.205+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~TVI8LA6NFHK3.0 size=1048576 2023-12-03T11:48:45.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.206+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~TVI8LA6NFHK3.1 size=702934 2023-12-03T11:48:45.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.206+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=TVI8LA6NFHK3 tag=GLWA4FARL01X type=write size=2800086 2023-12-03T11:48:45.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.207+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.0 size=1048576 2023-12-03T11:48:45.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.208+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.1 size=1048576 2023-12-03T11:48:45.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.209+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~WVOBOKSUTG11.2 size=575396 2023-12-03T11:48:45.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.211+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=WVOBOKSUTG11 tag=3QYG0QVAR7WB type=write size=3721124 2023-12-03T11:48:45.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.213+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~LQDHB0FEE945.0 size=1048576 2023-12-03T11:48:45.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.214+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~LQDHB0FEE945.1 size=448545 2023-12-03T11:48:45.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.214+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=LQDHB0FEE945 tag=5RRSFVWMO5HJ type=write size=2545697 2023-12-03T11:48:45.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.215+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=T42UB33MFDP0 type=del 2023-12-03T11:48:45.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.216+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=8G2AZDLM2Q3G type=del size=2194783 2023-12-03T11:48:45.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.217+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.0 size=1048576 2023-12-03T11:48:45.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.217+0000 7f24574f5780 1 < canceled part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.1 size=1048576 2023-12-03T11:48:45.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.218+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~F26ELAS2DOGX.2 size=533895 2023-12-03T11:48:45.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.220+0000 7f24574f5780 1 < canceled multipart upload key=QVLPCWC4T8N8 upload=F26ELAS2DOGX tag=GQE9B7L4SWQW type=write size=3679623 2023-12-03T11:48:45.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.222+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~N08W36GO4MO1.0 size=709371 2023-12-03T11:48:45.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.222+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=N08W36GO4MO1 tag=U15083JYQOWJ type=write size=1757947 2023-12-03T11:48:45.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.223+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=KVNQ4N1SQQOS type=del 2023-12-03T11:48:45.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.225+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~XUGVCCL9WVED.0 size=1048576 2023-12-03T11:48:45.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.226+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~XUGVCCL9WVED.1 size=377801 2023-12-03T11:48:45.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.226+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=XUGVCCL9WVED tag=AOIEU5NBIP1D type=write size=2474953 2023-12-03T11:48:45.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.227+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~2CD24K3OQIUD.0 size=1048576 2023-12-03T11:48:45.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.227+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~2CD24K3OQIUD.1 size=157813 2023-12-03T11:48:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.228+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=2CD24K3OQIUD tag=N5ZILBWRX1NA type=write size=2254965 2023-12-03T11:48:45.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.230+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=MIY1IC3PKKSV type=write size=851536 2023-12-03T11:48:45.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.231+0000 7f24574f5780 -1 < failed to complete operation key=S7Q5TCM499H1 tag=3DEQ7OB14S4N type=del size=816941: (22) Invalid argument 2023-12-03T11:48:45.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.233+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~NI8EF91GHC3H.0 size=1048576 2023-12-03T11:48:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.234+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~NI8EF91GHC3H.1 size=30363 2023-12-03T11:48:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.234+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=NI8EF91GHC3H tag=GIYPQ34ZL4NU type=write size=2127515 2023-12-03T11:48:45.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.235+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~HOGX3O7CZW7D.0 size=1048576 2023-12-03T11:48:45.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.235+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~HOGX3O7CZW7D.1 size=778161 2023-12-03T11:48:45.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.237+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=HOGX3O7CZW7D tag=G3O554CDJ32K type=write size=2875313 2023-12-03T11:48:45.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.239+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=I9WR9S39H6BZ type=del size=2702200 2023-12-03T11:48:45.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.240+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=3PD8W1060O55 type=del 2023-12-03T11:48:45.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.241+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=V3JDXAB443BI type=del size=4155538 2023-12-03T11:48:45.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.243+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=CQXU1HJ2A76L type=del size=3776437 2023-12-03T11:48:45.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.245+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=7KA26E47IW2Z type=del size=599023 2023-12-03T11:48:45.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.246+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=SNQ0VFFPWKP6 type=del 2023-12-03T11:48:45.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.247+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=4TP1MWKHLV0K type=del size=471782 2023-12-03T11:48:45.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.248+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~IFJCELMFWWSY.0 size=853330 2023-12-03T11:48:45.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.250+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=IFJCELMFWWSY tag=7D0VJAW18EQ1 type=write size=1901906 2023-12-03T11:48:45.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.252+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=UVKKG35R1882 type=del size=3762691 2023-12-03T11:48:45.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.253+0000 7f24574f5780 1 < canceled operation key=QVLPCWC4T8N8 tag=V2TRJOJCGTIW type=del size=3252044 2023-12-03T11:48:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.254+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=E3M4Z28IK8DT type=del size=1914887 2023-12-03T11:48:45.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.256+0000 7f24574f5780 1 < suggested update operation key=LS4OEAGZWQXC tag=HO79DZ811M8K type=write size=853732 2023-12-03T11:48:45.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.258+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~M80JXWKDJVEZ.0 size=1048576 2023-12-03T11:48:45.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.259+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~M80JXWKDJVEZ.1 size=298523 2023-12-03T11:48:45.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.259+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=M80JXWKDJVEZ tag=AROQDHHGP23S type=write size=2395675 2023-12-03T11:48:45.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.261+0000 7f24574f5780 1 < completed write operation key=ON2E6H35N9FJ tag=1S4U6ZJDOD1T type=write size=255138 2023-12-03T11:48:45.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.262+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=0S6OC41RG28Z type=write size=684576 2023-12-03T11:48:45.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.263+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=E25PRFFD0CBE type=del 2023-12-03T11:48:45.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.264+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=XWDDJFMR5A8M type=del size=3662654 2023-12-03T11:48:45.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.265+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~TVI8LA6NFHK3.0 size=1048576 2023-12-03T11:48:45.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.266+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~TVI8LA6NFHK3.1 size=702934 2023-12-03T11:48:45.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.268+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=TVI8LA6NFHK3 tag=GLWA4FARL01X type=write size=2800086 2023-12-03T11:48:45.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.270+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~UQWZK938Z3TM.0 size=1048576 2023-12-03T11:48:45.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.271+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~UQWZK938Z3TM.1 size=675965 2023-12-03T11:48:45.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.271+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=UQWZK938Z3TM tag=LGAFY1GP5P26 type=write size=2773117 2023-12-03T11:48:45.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.272+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~LQDHB0FEE945.0 size=1048576 2023-12-03T11:48:45.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.273+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~LQDHB0FEE945.1 size=448545 2023-12-03T11:48:45.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.275+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=LQDHB0FEE945 tag=5RRSFVWMO5HJ type=write size=2545697 2023-12-03T11:48:45.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.277+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=JKBPMB2QFWC2 type=del size=1897841 2023-12-03T11:48:45.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.278+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=8G2AZDLM2Q3G type=del 2023-12-03T11:48:45.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.280+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~UDP85DMJETUA.0 size=1048576 2023-12-03T11:48:45.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.281+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~UDP85DMJETUA.1 size=569014 2023-12-03T11:48:45.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.281+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=UDP85DMJETUA tag=426TC8LBQDEG type=write size=2666166 2023-12-03T11:48:45.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.284+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~N08W36GO4MO1.0 size=709371 2023-12-03T11:48:45.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.285+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=N08W36GO4MO1 tag=U15083JYQOWJ type=write size=1757947 2023-12-03T11:48:45.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.287+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.0 size=1048576 2023-12-03T11:48:45.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.288+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.1 size=1048576 2023-12-03T11:48:45.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.290+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.2 size=213712 2023-12-03T11:48:45.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.290+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=9IK2K4MENO08 tag=N59HVJP1BEMO type=write size=3359440 2023-12-03T11:48:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.291+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~XUGVCCL9WVED.0 size=1048576 2023-12-03T11:48:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.293+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~XUGVCCL9WVED.1 size=377801 2023-12-03T11:48:45.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.295+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=XUGVCCL9WVED tag=AOIEU5NBIP1D type=write size=2474953 2023-12-03T11:48:45.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.296+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=5Z8M7EB9KQD1 type=del size=2291082 2023-12-03T11:48:45.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.297+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=MIY1IC3PKKSV type=write size=851536 2023-12-03T11:48:45.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.298+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=IKKARK7B0Y3T type=del size=3154661 2023-12-03T11:48:45.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.299+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~NI8EF91GHC3H.0 size=1048576 2023-12-03T11:48:45.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.300+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~NI8EF91GHC3H.1 size=30363 2023-12-03T11:48:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.302+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=NI8EF91GHC3H tag=GIYPQ34ZL4NU type=write size=2127515 2023-12-03T11:48:45.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.303+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=Z2ZJS6YVHRO9 type=del size=2076365 2023-12-03T11:48:45.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.304+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=I9WR9S39H6BZ type=del 2023-12-03T11:48:45.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.305+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=ZK2EWQG09IJW type=del size=3489087 2023-12-03T11:48:45.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.307+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=V3JDXAB443BI type=del 2023-12-03T11:48:45.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.308+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=7CSLUAFE9EV6 type=del size=2217663 2023-12-03T11:48:45.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.309+0000 7f24574f5780 1 < canceled operation key=DA6ZXCLKTC1L tag=7KA26E47IW2Z type=del size=599023 2023-12-03T11:48:45.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.311+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~YN8O546CGV1E.0 size=367061 2023-12-03T11:48:45.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.311+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=YN8O546CGV1E tag=IYL96FDX72P9 type=write size=1415637 2023-12-03T11:48:45.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.312+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=4TP1MWKHLV0K type=del 2023-12-03T11:48:45.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.314+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~8CPL6IQKWGRO.0 size=753335 2023-12-03T11:48:45.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.314+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=8CPL6IQKWGRO tag=J935N08NVEMH type=write size=1801911 2023-12-03T11:48:45.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.316+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=UVKKG35R1882 type=del 2023-12-03T11:48:45.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.317+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=FLFM5LVHMTBK type=del size=2849320 2023-12-03T11:48:45.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.319+0000 7f24574f5780 1 < suggested remove operation key=1V39QA10M1HY tag=E3M4Z28IK8DT type=del 2023-12-03T11:48:45.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.320+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=29BX7Q5WI3RF type=write size=1007001 2023-12-03T11:48:45.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.322+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~M80JXWKDJVEZ.0 size=1048576 2023-12-03T11:48:45.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.323+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~M80JXWKDJVEZ.1 size=298523 2023-12-03T11:48:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.325+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=M80JXWKDJVEZ tag=AROQDHHGP23S type=write size=2395675 2023-12-03T11:48:45.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.326+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=CI6B0T0Q8B4H type=del size=658584 2023-12-03T11:48:45.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.327+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=0S6OC41RG28Z type=write size=684576 2023-12-03T11:48:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.329+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~GJK8EGPAXRCH.0 size=1048576 2023-12-03T11:48:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.331+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~GJK8EGPAXRCH.1 size=335413 2023-12-03T11:48:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.331+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=GJK8EGPAXRCH tag=CKS218LW0TN2 type=write size=2432565 2023-12-03T11:48:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.332+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=XWDDJFMR5A8M type=del 2023-12-03T11:48:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.333+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=FTJ7NV33OZR1 type=del size=1307782 2023-12-03T11:48:45.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.335+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~UQWZK938Z3TM.0 size=1048576 2023-12-03T11:48:45.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.336+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~UQWZK938Z3TM.1 size=675965 2023-12-03T11:48:45.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.338+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=UQWZK938Z3TM tag=LGAFY1GP5P26 type=write size=2773117 2023-12-03T11:48:45.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.340+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.0 size=1048576 2023-12-03T11:48:45.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.341+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.1 size=1048576 2023-12-03T11:48:45.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.342+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.2 size=885933 2023-12-03T11:48:45.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.342+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=P6GSQ31PM5CN tag=LTC77URGR9O8 type=write size=4031661 2023-12-03T11:48:45.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.343+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=JKBPMB2QFWC2 type=del 2023-12-03T11:48:45.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.344+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=2EGUSSSCB5OL type=del size=1320379 2023-12-03T11:48:45.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.345+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~UDP85DMJETUA.0 size=1048576 2023-12-03T11:48:45.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.346+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~UDP85DMJETUA.1 size=569014 2023-12-03T11:48:45.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.348+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=UDP85DMJETUA tag=426TC8LBQDEG type=write size=2666166 2023-12-03T11:48:45.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.349+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=R0ZDF5WTGVX1 type=del size=995987 2023-12-03T11:48:45.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.351+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.0 size=1048576 2023-12-03T11:48:45.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.352+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.1 size=1048576 2023-12-03T11:48:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.353+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~9IK2K4MENO08.2 size=213712 2023-12-03T11:48:45.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.355+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=9IK2K4MENO08 tag=N59HVJP1BEMO type=write size=3359440 2023-12-03T11:48:45.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.356+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=2662Y0MWONAB type=del size=2625392 2023-12-03T11:48:45.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.357+0000 7f24574f5780 -1 < failed to complete operation key=1V39QA10M1HY tag=5Z8M7EB9KQD1 type=del size=2291082: (22) Invalid argument 2023-12-03T11:48:45.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.358+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=ELU264DR9OXJ type=del size=3595037 2023-12-03T11:48:45.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.359+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=IKKARK7B0Y3T type=del 2023-12-03T11:48:45.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.362+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~CE5NAQ2EITER.0 size=420809 2023-12-03T11:48:45.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.362+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=CE5NAQ2EITER tag=U261UZ4QJ2BN type=write size=1469385 2023-12-03T11:48:45.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.363+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=Z2ZJS6YVHRO9 type=del 2023-12-03T11:48:45.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.364+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=ISXLD03SKFRK type=del size=2377868 2023-12-03T11:48:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.365+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=ZK2EWQG09IJW type=del 2023-12-03T11:48:45.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.368+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.0 size=1048576 2023-12-03T11:48:45.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.369+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.1 size=1048576 2023-12-03T11:48:45.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.370+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.2 size=754699 2023-12-03T11:48:45.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.370+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=BPN0HJUUCH1N tag=MCE1FT7FREA0 type=write size=3900427 2023-12-03T11:48:45.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.371+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=7CSLUAFE9EV6 type=del 2023-12-03T11:48:45.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.372+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=KO42LCU0J544 type=del size=3346588 2023-12-03T11:48:45.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.373+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~YN8O546CGV1E.0 size=367061 2023-12-03T11:48:45.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.375+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=YN8O546CGV1E tag=IYL96FDX72P9 type=write size=1415637 2023-12-03T11:48:45.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.376+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=P5J88RK17OKX type=del size=1924460 2023-12-03T11:48:45.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.377+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~8CPL6IQKWGRO.0 size=753335 2023-12-03T11:48:45.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.379+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=8CPL6IQKWGRO tag=J935N08NVEMH type=write size=1801911 2023-12-03T11:48:45.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.381+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=WOG08CTXG2XP type=del size=156761 2023-12-03T11:48:45.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.382+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=FLFM5LVHMTBK type=del 2023-12-03T11:48:45.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.383+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=KLU9YSIR0N1T type=del size=2765512 2023-12-03T11:48:45.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.384+0000 7f24574f5780 1 < canceled operation key=PS1OYWI44L80 tag=29BX7Q5WI3RF type=write size=1007001 2023-12-03T11:48:45.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.386+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.0 size=1048576 2023-12-03T11:48:45.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.387+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.1 size=1048576 2023-12-03T11:48:45.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.388+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.2 size=1043838 2023-12-03T11:48:45.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.388+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=5VDLV24PH8TJ tag=LDGHKMJZJJ11 type=write size=4189566 2023-12-03T11:48:45.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.390+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=CI6B0T0Q8B4H type=del 2023-12-03T11:48:45.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.392+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=4S9Y3F9OUNL8 type=del size=2936094 2023-12-03T11:48:45.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.393+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~GJK8EGPAXRCH.0 size=1048576 2023-12-03T11:48:45.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.394+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~GJK8EGPAXRCH.1 size=335413 2023-12-03T11:48:45.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.397+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=GJK8EGPAXRCH tag=CKS218LW0TN2 type=write size=2432565 2023-12-03T11:48:45.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.398+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=IYKRQF7268Y6 type=write size=639698 2023-12-03T11:48:45.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.399+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=FTJ7NV33OZR1 type=del 2023-12-03T11:48:45.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.400+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=WW2J2JAKPC2F type=del size=2399683 2023-12-03T11:48:45.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.402+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.0 size=1048576 2023-12-03T11:48:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.404+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.1 size=1048576 2023-12-03T11:48:45.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.405+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~P6GSQ31PM5CN.2 size=885933 2023-12-03T11:48:45.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.408+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=P6GSQ31PM5CN tag=LTC77URGR9O8 type=write size=4031661 2023-12-03T11:48:45.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.409+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=EEDIHKCUQZAM type=write size=940646 2023-12-03T11:48:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.411+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=2EGUSSSCB5OL type=del 2023-12-03T11:48:45.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.413+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~8C0DLWGSIDWK.0 size=315877 2023-12-03T11:48:45.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.413+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=8C0DLWGSIDWK tag=JZ4H8IN2IVIY type=write size=1364453 2023-12-03T11:48:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.414+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=R0ZDF5WTGVX1 type=del 2023-12-03T11:48:45.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.415+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=WFEG3UDPMUF7 type=del size=1594603 2023-12-03T11:48:45.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.418+0000 7f24574f5780 1 < suggested remove operation key=MY3K9M4QRS3L tag=2662Y0MWONAB type=del 2023-12-03T11:48:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.419+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=NRCQ2WLMXUGI type=del size=1199240 2023-12-03T11:48:45.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.420+0000 7f24574f5780 1 < canceled operation key=WZMWCSAZX8RB tag=ELU264DR9OXJ type=del size=3595037 2023-12-03T11:48:45.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.422+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=8A26BUNEX0LF type=write size=770258 2023-12-03T11:48:45.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.423+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~CE5NAQ2EITER.0 size=420809 2023-12-03T11:48:45.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.425+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=CE5NAQ2EITER tag=U261UZ4QJ2BN type=write size=1469385 2023-12-03T11:48:45.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.427+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.0 size=1048576 2023-12-03T11:48:45.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.428+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.1 size=1048576 2023-12-03T11:48:45.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.429+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.2 size=4504 2023-12-03T11:48:45.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.429+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=WR37P2LWCBUG tag=WYY6I6JF3KV3 type=write size=3150232 2023-12-03T11:48:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.430+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=ISXLD03SKFRK type=del 2023-12-03T11:48:45.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.432+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=ZBVKQ3PLEYC7 type=del size=3649477 2023-12-03T11:48:45.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.433+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.0 size=1048576 2023-12-03T11:48:45.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.434+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.1 size=1048576 2023-12-03T11:48:45.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.435+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~BPN0HJUUCH1N.2 size=754699 2023-12-03T11:48:45.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.437+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=BPN0HJUUCH1N tag=MCE1FT7FREA0 type=write size=3900427 2023-12-03T11:48:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.440+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.0 size=1048576 2023-12-03T11:48:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.441+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.1 size=1048576 2023-12-03T11:48:45.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.442+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.2 size=54207 2023-12-03T11:48:45.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.442+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=8UZAB7JL7AYN tag=WYN7AQ45R06W type=write size=3199935 2023-12-03T11:48:45.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.443+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=KO42LCU0J544 type=del 2023-12-03T11:48:45.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.444+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=ZCKU97QFNE7V type=del size=69064 2023-12-03T11:48:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.446+0000 7f24574f5780 1 < suggested remove operation key=35U7YCDYMX46 tag=P5J88RK17OKX type=del 2023-12-03T11:48:45.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.448+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=KO064YMT301I type=del size=485685 2023-12-03T11:48:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.449+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=WOG08CTXG2XP type=del 2023-12-03T11:48:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.450+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=1PGTX6HNFIGS type=del size=2394810 2023-12-03T11:48:45.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.451+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=KLU9YSIR0N1T type=del 2023-12-03T11:48:45.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.453+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.0 size=1048576 2023-12-03T11:48:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.454+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.1 size=1048576 2023-12-03T11:48:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.455+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.2 size=900663 2023-12-03T11:48:45.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.455+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=EH1RJUIWH9L9 tag=WBCVKNDZP90M type=write size=4046391 2023-12-03T11:48:45.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.456+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.0 size=1048576 2023-12-03T11:48:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.457+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.1 size=1048576 2023-12-03T11:48:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.458+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~5VDLV24PH8TJ.2 size=1043838 2023-12-03T11:48:45.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.460+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=5VDLV24PH8TJ tag=LDGHKMJZJJ11 type=write size=4189566 2023-12-03T11:48:45.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.461+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=TBFJBUYNIW07 type=write size=710888 2023-12-03T11:48:45.464 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.462+0000 7f24574f5780 1 < canceled operation key=SC8UOVLQQ0VD tag=4S9Y3F9OUNL8 type=del size=2936094 2023-12-03T11:48:45.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.464+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=LQD1DDW0V8OW type=write size=895893 2023-12-03T11:48:45.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.465+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=IYKRQF7268Y6 type=write size=639698 2023-12-03T11:48:45.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.467+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~7KJQRHIT3L3Z.0 size=1048576 2023-12-03T11:48:45.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.468+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~7KJQRHIT3L3Z.1 size=683235 2023-12-03T11:48:45.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.468+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=7KJQRHIT3L3Z tag=6VSZNFHYO93M type=write size=2780387 2023-12-03T11:48:45.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.469+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=WW2J2JAKPC2F type=del 2023-12-03T11:48:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.470+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=RUX060KS925S type=write size=868631 2023-12-03T11:48:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.471+0000 7f24574f5780 -1 < failed to complete operation key=35U7YCDYMX46 tag=EEDIHKCUQZAM type=write size=940646: (22) Invalid argument 2023-12-03T11:48:45.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.473+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~AXOROQL23MCM.0 size=1048576 2023-12-03T11:48:45.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.474+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~AXOROQL23MCM.1 size=454686 2023-12-03T11:48:45.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.474+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=AXOROQL23MCM tag=3RGXVELOX71M type=write size=2551838 2023-12-03T11:48:45.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.475+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~8C0DLWGSIDWK.0 size=315877 2023-12-03T11:48:45.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.477+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=8C0DLWGSIDWK tag=JZ4H8IN2IVIY type=write size=1364453 2023-12-03T11:48:45.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.478+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=L9FL8Q1F4HS9 type=del size=1501353 2023-12-03T11:48:45.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.480+0000 7f24574f5780 -1 < failed to complete operation key=MY3K9M4QRS3L tag=WFEG3UDPMUF7 type=del size=1594603: (22) Invalid argument 2023-12-03T11:48:45.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.481+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=UXICM01OF7OV type=del size=1107413 2023-12-03T11:48:45.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.482+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=NRCQ2WLMXUGI type=del 2023-12-03T11:48:45.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.483+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=JBWROIEIUSQW type=del size=1467350 2023-12-03T11:48:45.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.484+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=8A26BUNEX0LF type=write size=770258 2023-12-03T11:48:45.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.485+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=OQZRX6GN2QBJ type=del size=734549 2023-12-03T11:48:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.486+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.0 size=1048576 2023-12-03T11:48:45.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.487+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.1 size=1048576 2023-12-03T11:48:45.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.487+0000 7f24574f5780 1 < canceled part key=_multipart_PS1OYWI44L80.2~WR37P2LWCBUG.2 size=4504 2023-12-03T11:48:45.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.489+0000 7f24574f5780 1 < canceled multipart upload key=PS1OYWI44L80 upload=WR37P2LWCBUG tag=WYY6I6JF3KV3 type=write size=3150232 2023-12-03T11:48:45.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.490+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=F6CI5BSWKBQT type=del size=3660882 2023-12-03T11:48:45.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.492+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=ZBVKQ3PLEYC7 type=del 2023-12-03T11:48:45.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.493+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=MZ51GNYUTN5K type=write size=246091 2023-12-03T11:48:45.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.494+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.0 size=1048576 2023-12-03T11:48:45.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.495+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.1 size=1048576 2023-12-03T11:48:45.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.495+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~8UZAB7JL7AYN.2 size=54207 2023-12-03T11:48:45.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.497+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=8UZAB7JL7AYN tag=WYN7AQ45R06W type=write size=3199935 2023-12-03T11:48:45.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.498+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=PBA3K4NA1MTS type=del size=1963382 2023-12-03T11:48:45.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.500+0000 7f24574f5780 1 < suggested update operation key=PGTGVLK4DBN0 tag=ZCKU97QFNE7V type=del size=69064 2023-12-03T11:48:45.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.502+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=XLPBFJLF4US7 type=del size=3983573 2023-12-03T11:48:45.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.503+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=KO064YMT301I type=del 2023-12-03T11:48:45.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.505+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~1PA07BI7OLPU.0 size=592401 2023-12-03T11:48:45.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.505+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=1PA07BI7OLPU tag=LK763CZFTZN0 type=write size=1640977 2023-12-03T11:48:45.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.507+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=1PGTX6HNFIGS type=del 2023-12-03T11:48:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.508+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=GHKB2DJS5Z8C type=del size=2131395 2023-12-03T11:48:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.508+0000 7f24574f5780 1 < canceled part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.0 size=1048576 2023-12-03T11:48:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.509+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.1 size=1048576 2023-12-03T11:48:45.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.510+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~EH1RJUIWH9L9.2 size=900663 2023-12-03T11:48:45.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.512+0000 7f24574f5780 1 < canceled multipart upload key=ON2E6H35N9FJ upload=EH1RJUIWH9L9 tag=WBCVKNDZP90M type=write size=4046391 2023-12-03T11:48:45.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.513+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=YI6Q6PFB7PA0 type=write size=908409 2023-12-03T11:48:45.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.515+0000 7f24574f5780 1 < completed write operation key=BV8C0VSJ1GTW tag=TBFJBUYNIW07 type=write size=710888 2023-12-03T11:48:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.518+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~BSRF4N95EZ8S.0 size=1048576 2023-12-03T11:48:45.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.521+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~BSRF4N95EZ8S.1 size=942840 2023-12-03T11:48:45.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.521+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=BSRF4N95EZ8S tag=0SQ2KI6916UB type=write size=3039992 2023-12-03T11:48:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.522+0000 7f24574f5780 1 < completed write operation key=S73M6UFMX5NH tag=LQD1DDW0V8OW type=write size=895893 2023-12-03T11:48:45.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.524+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=A0PRNRHL2LP7 type=del size=835096 2023-12-03T11:48:45.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.525+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~7KJQRHIT3L3Z.0 size=1048576 2023-12-03T11:48:45.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.526+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~7KJQRHIT3L3Z.1 size=683235 2023-12-03T11:48:45.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.528+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=7KJQRHIT3L3Z tag=6VSZNFHYO93M type=write size=2780387 2023-12-03T11:48:45.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.529+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=JJZ3N0HGQPEM type=del size=3084645 2023-12-03T11:48:45.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.531+0000 7f24574f5780 1 < completed write operation key=6WY91DFIRUN8 tag=RUX060KS925S type=write size=868631 2023-12-03T11:48:45.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.532+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~39WO3YLEYL6R.0 size=897865 2023-12-03T11:48:45.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.532+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=39WO3YLEYL6R tag=G9CEXNXKDWB3 type=write size=1946441 2023-12-03T11:48:45.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.534+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~AXOROQL23MCM.0 size=1048576 2023-12-03T11:48:45.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.535+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~AXOROQL23MCM.1 size=454686 2023-12-03T11:48:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.537+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=AXOROQL23MCM tag=3RGXVELOX71M type=write size=2551838 2023-12-03T11:48:45.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.539+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~9UX2OQZJAJC7.0 size=284466 2023-12-03T11:48:45.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.539+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=9UX2OQZJAJC7 tag=70U0SRERKNEO type=write size=1333042 2023-12-03T11:48:45.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.541+0000 7f24574f5780 1 < suggested update operation key=LS4OEAGZWQXC tag=L9FL8Q1F4HS9 type=del size=1501353 2023-12-03T11:48:45.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.542+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=1S1BIPQID07H type=del size=815999 2023-12-03T11:48:45.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.543+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=UXICM01OF7OV type=del 2023-12-03T11:48:45.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.544+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=0KDFDKPNZ74W type=write size=286496 2023-12-03T11:48:45.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.546+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=JBWROIEIUSQW type=del 2023-12-03T11:48:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.547+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=SLJYE7Q6MXYM type=del size=3673046 2023-12-03T11:48:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.548+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=OQZRX6GN2QBJ type=del 2023-12-03T11:48:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.550+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~Q0HRV40GD54C.0 size=787326 2023-12-03T11:48:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.550+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=Q0HRV40GD54C tag=QNV80C28HMQL type=write size=1835902 2023-12-03T11:48:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.552+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=F6CI5BSWKBQT type=del 2023-12-03T11:48:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.553+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=7UODA29G2RYE type=write size=379450 2023-12-03T11:48:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.554+0000 7f24574f5780 -1 < failed to complete operation key=PGTGVLK4DBN0 tag=MZ51GNYUTN5K type=write size=246091: (22) Invalid argument 2023-12-03T11:48:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.556+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~Z9NIZ47KG76A.0 size=262507 2023-12-03T11:48:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.556+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=Z9NIZ47KG76A tag=VWB8PGO31KKU type=write size=1311083 2023-12-03T11:48:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.557+0000 7f24574f5780 -1 < failed to complete operation key=PGTGVLK4DBN0 tag=PBA3K4NA1MTS type=del size=1963382: (22) Invalid argument 2023-12-03T11:48:45.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.558+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=OO6ZBWVY6OER type=del size=833676 2023-12-03T11:48:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.560+0000 7f24574f5780 1 < canceled operation key=S73M6UFMX5NH tag=XLPBFJLF4US7 type=del size=3983573 2023-12-03T11:48:45.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.561+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=2N3WX4LBQQB3 type=del size=3802072 2023-12-03T11:48:45.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.562+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~1PA07BI7OLPU.0 size=592401 2023-12-03T11:48:45.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.564+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=1PA07BI7OLPU tag=LK763CZFTZN0 type=write size=1640977 2023-12-03T11:48:45.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.565+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=KAC8LJK2KTIH type=del size=3105753 2023-12-03T11:48:45.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.566+0000 7f24574f5780 -1 < failed to complete operation key=DA6ZXCLKTC1L tag=GHKB2DJS5Z8C type=del size=2131395: (22) Invalid argument 2023-12-03T11:48:45.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.567+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=0NCYS9QEAFHP type=del size=619027 2023-12-03T11:48:45.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.568+0000 7f24574f5780 1 < completed write operation key=WZMWCSAZX8RB tag=YI6Q6PFB7PA0 type=write size=908409 2023-12-03T11:48:45.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.569+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=NUX8RVGXYTTQ type=del size=3231119 2023-12-03T11:48:45.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.570+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~BSRF4N95EZ8S.0 size=1048576 2023-12-03T11:48:45.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.570+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~BSRF4N95EZ8S.1 size=942840 2023-12-03T11:48:45.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.572+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=BSRF4N95EZ8S tag=0SQ2KI6916UB type=write size=3039992 2023-12-03T11:48:45.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.574+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=FEU2054QQTMR type=del size=2278971 2023-12-03T11:48:45.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.576+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=A0PRNRHL2LP7 type=del 2023-12-03T11:48:45.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.578+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~468FUYJEMRSL.0 size=573490 2023-12-03T11:48:45.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.578+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=468FUYJEMRSL tag=6MI5YIQE9QIS type=write size=1622066 2023-12-03T11:48:45.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.580+0000 7f24574f5780 1 < canceled operation key=S73M6UFMX5NH tag=JJZ3N0HGQPEM type=del size=3084645 2023-12-03T11:48:45.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.582+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~LYPP7MS3TNJ7.0 size=352036 2023-12-03T11:48:45.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.582+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=LYPP7MS3TNJ7 tag=SF12IFE4QL2K type=write size=1400612 2023-12-03T11:48:45.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.583+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~39WO3YLEYL6R.0 size=897865 2023-12-03T11:48:45.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.585+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=39WO3YLEYL6R tag=G9CEXNXKDWB3 type=write size=1946441 2023-12-03T11:48:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.588+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.0 size=1048576 2023-12-03T11:48:45.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.589+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.1 size=1048576 2023-12-03T11:48:45.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.589+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.2 size=975129 2023-12-03T11:48:45.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.589+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=71FX7D9NB6NN tag=7W3JOVJMMV3S type=write size=4120857 2023-12-03T11:48:45.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.591+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~9UX2OQZJAJC7.0 size=284466 2023-12-03T11:48:45.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.593+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=9UX2OQZJAJC7 tag=70U0SRERKNEO type=write size=1333042 2023-12-03T11:48:45.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.594+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=KY2FSCA08SJ9 type=del size=3199075 2023-12-03T11:48:45.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.595+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=1S1BIPQID07H type=del 2023-12-03T11:48:45.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.597+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~N4RW0NP3N5A2.0 size=621927 2023-12-03T11:48:45.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.597+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=N4RW0NP3N5A2 tag=UJG04HD7EOQF type=write size=1670503 2023-12-03T11:48:45.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.598+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=0KDFDKPNZ74W type=write size=286496 2023-12-03T11:48:45.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.599+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=OI8FMX9BAZLZ type=del size=1949145 2023-12-03T11:48:45.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.602+0000 7f24574f5780 1 < suggested update operation key=A4LOQE7VRDN7 tag=SLJYE7Q6MXYM type=del size=3673046 2023-12-03T11:48:45.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.603+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=XVIOYKHMWT7E type=del size=269894 2023-12-03T11:48:45.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.604+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~Q0HRV40GD54C.0 size=787326 2023-12-03T11:48:45.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.606+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=Q0HRV40GD54C tag=QNV80C28HMQL type=write size=1835902 2023-12-03T11:48:45.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.608+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~6J4YMTVEAOU9.0 size=716266 2023-12-03T11:48:45.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.608+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=6J4YMTVEAOU9 tag=YM3RILR3F9CV type=write size=1764842 2023-12-03T11:48:45.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.609+0000 7f24574f5780 1 < canceled operation key=XZFHFS76WNR4 tag=7UODA29G2RYE type=write size=379450 2023-12-03T11:48:45.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.611+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=4GHEHZOEUCCN type=del size=3560767 2023-12-03T11:48:45.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.612+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~Z9NIZ47KG76A.0 size=262507 2023-12-03T11:48:45.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.614+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=Z9NIZ47KG76A tag=VWB8PGO31KKU type=write size=1311083 2023-12-03T11:48:45.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.615+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=GPXO5BOY2JV0 type=del size=3087435 2023-12-03T11:48:45.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.616+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=OO6ZBWVY6OER type=del 2023-12-03T11:48:45.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.618+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.0 size=1048576 2023-12-03T11:48:45.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.619+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.1 size=1048576 2023-12-03T11:48:45.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.620+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.2 size=719034 2023-12-03T11:48:45.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.620+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=V1JK80WGFVEC tag=XTGLM4QPDJO7 type=write size=3864762 2023-12-03T11:48:45.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.622+0000 7f24574f5780 1 < suggested remove operation key=9LKFY1U0JRBQ tag=2N3WX4LBQQB3 type=del 2023-12-03T11:48:45.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.623+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=RCZTZX8525VX type=write size=95449 2023-12-03T11:48:45.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.625+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=KAC8LJK2KTIH type=del 2023-12-03T11:48:45.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.626+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=WI9FMS9E5ZEU type=del size=3168657 2023-12-03T11:48:45.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.627+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=0NCYS9QEAFHP type=del 2023-12-03T11:48:45.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.629+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~QNK2K0N9I2PJ.0 size=1048576 2023-12-03T11:48:45.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.631+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~QNK2K0N9I2PJ.1 size=476314 2023-12-03T11:48:45.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.631+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=QNK2K0N9I2PJ tag=15XV8SB32R5V type=write size=2573466 2023-12-03T11:48:45.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.632+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=NUX8RVGXYTTQ type=del 2023-12-03T11:48:45.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.634+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~MC0S80WAD09Y.0 size=969817 2023-12-03T11:48:45.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.634+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=MC0S80WAD09Y tag=L681BDEX1W77 type=write size=2018393 2023-12-03T11:48:45.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.637+0000 7f24574f5780 1 < suggested update operation key=WZMWCSAZX8RB tag=FEU2054QQTMR type=del size=2278971 2023-12-03T11:48:45.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.638+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=X2TP3TSM5HUL type=del size=3414 2023-12-03T11:48:45.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.639+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~468FUYJEMRSL.0 size=573490 2023-12-03T11:48:45.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.641+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=468FUYJEMRSL tag=6MI5YIQE9QIS type=write size=1622066 2023-12-03T11:48:45.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.642+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=YCKRBHBM9GU1 type=del size=146636 2023-12-03T11:48:45.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.643+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~LYPP7MS3TNJ7.0 size=352036 2023-12-03T11:48:45.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.645+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=LYPP7MS3TNJ7 tag=SF12IFE4QL2K type=write size=1400612 2023-12-03T11:48:45.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.647+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~48XIIPQSB1N2.0 size=1048576 2023-12-03T11:48:45.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.648+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~48XIIPQSB1N2.1 size=767060 2023-12-03T11:48:45.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.648+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=48XIIPQSB1N2 tag=HYGITVW9ZI1Z type=write size=2864212 2023-12-03T11:48:45.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.649+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.0 size=1048576 2023-12-03T11:48:45.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.650+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.1 size=1048576 2023-12-03T11:48:45.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.653+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~71FX7D9NB6NN.2 size=975129 2023-12-03T11:48:45.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.655+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=71FX7D9NB6NN tag=7W3JOVJMMV3S type=write size=4120857 2023-12-03T11:48:45.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.656+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=80BJW95SD415 type=del size=2376087 2023-12-03T11:48:45.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.657+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=KY2FSCA08SJ9 type=del 2023-12-03T11:48:45.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.659+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~WERKX6QM2AP4.0 size=1048576 2023-12-03T11:48:45.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.660+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~WERKX6QM2AP4.1 size=429895 2023-12-03T11:48:45.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.660+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=WERKX6QM2AP4 tag=ELO3JTA6J0VG type=write size=2527047 2023-12-03T11:48:45.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.661+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~N4RW0NP3N5A2.0 size=621927 2023-12-03T11:48:45.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.663+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=N4RW0NP3N5A2 tag=UJG04HD7EOQF type=write size=1670503 2023-12-03T11:48:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.664+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=JU64O7FQJC2W type=del size=1253625 2023-12-03T11:48:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.666+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=OI8FMX9BAZLZ type=del 2023-12-03T11:48:45.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.667+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=04MNM0VW8B6Q type=del size=657403 2023-12-03T11:48:45.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.669+0000 7f24574f5780 1 < suggested update operation key=A4LOQE7VRDN7 tag=XVIOYKHMWT7E type=del size=269894 2023-12-03T11:48:45.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.671+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~8I4Y72ZOQ9C5.0 size=1048576 2023-12-03T11:48:45.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.672+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~8I4Y72ZOQ9C5.1 size=45573 2023-12-03T11:48:45.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.672+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=8I4Y72ZOQ9C5 tag=4QMYSMQ5F6BU type=write size=2142725 2023-12-03T11:48:45.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.673+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~6J4YMTVEAOU9.0 size=716266 2023-12-03T11:48:45.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.675+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=6J4YMTVEAOU9 tag=YM3RILR3F9CV type=write size=1764842 2023-12-03T11:48:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.677+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~Y97S9M9XUIV3.0 size=1048576 2023-12-03T11:48:45.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.679+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~Y97S9M9XUIV3.1 size=239995 2023-12-03T11:48:45.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.679+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=Y97S9M9XUIV3 tag=TZWK0UZHKHKZ type=write size=2337147 2023-12-03T11:48:45.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.680+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=4GHEHZOEUCCN type=del 2023-12-03T11:48:45.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.682+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~IPD9JXCWQ3YM.0 size=795905 2023-12-03T11:48:45.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.682+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=IPD9JXCWQ3YM tag=TSOR1C9N8DPB type=write size=1844481 2023-12-03T11:48:45.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.683+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=GPXO5BOY2JV0 type=del 2023-12-03T11:48:45.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.685+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~99W0P3OO163E.0 size=1048576 2023-12-03T11:48:45.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.686+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~99W0P3OO163E.1 size=897369 2023-12-03T11:48:45.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.686+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=99W0P3OO163E tag=XX635S81IP4I type=write size=2994521 2023-12-03T11:48:45.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.687+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.0 size=1048576 2023-12-03T11:48:45.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.688+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.1 size=1048576 2023-12-03T11:48:45.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.689+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~V1JK80WGFVEC.2 size=719034 2023-12-03T11:48:45.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.691+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=V1JK80WGFVEC tag=XTGLM4QPDJO7 type=write size=3864762 2023-12-03T11:48:45.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.695+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.0 size=1048576 2023-12-03T11:48:45.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.696+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.1 size=1048576 2023-12-03T11:48:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.697+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.2 size=453332 2023-12-03T11:48:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.697+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=T7WSZNQA3SSZ tag=PGPTCPIYPQIW type=write size=3599060 2023-12-03T11:48:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.698+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=RCZTZX8525VX type=write size=95449 2023-12-03T11:48:45.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.699+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=R82LP2GZFYMZ type=del size=966692 2023-12-03T11:48:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.700+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=WI9FMS9E5ZEU type=del size=3168657 2023-12-03T11:48:45.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.702+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~H0V7MHJ28YZ3.0 size=248878 2023-12-03T11:48:45.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.702+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=H0V7MHJ28YZ3 tag=VOIG5UP2US79 type=write size=1297454 2023-12-03T11:48:45.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.703+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~QNK2K0N9I2PJ.0 size=1048576 2023-12-03T11:48:45.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.705+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~QNK2K0N9I2PJ.1 size=476314 2023-12-03T11:48:45.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.706+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=QNK2K0N9I2PJ tag=15XV8SB32R5V type=write size=2573466 2023-12-03T11:48:45.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.708+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.0 size=1048576 2023-12-03T11:48:45.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.709+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.1 size=1048576 2023-12-03T11:48:45.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.710+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.2 size=591890 2023-12-03T11:48:45.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.710+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=QAAQTRK19Y0C tag=Q9HVW00KFYOM type=write size=3737618 2023-12-03T11:48:45.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.712+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~MC0S80WAD09Y.0 size=969817 2023-12-03T11:48:45.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.713+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=MC0S80WAD09Y tag=L681BDEX1W77 type=write size=2018393 2023-12-03T11:48:45.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.715+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=RE0IRRM6J357 type=del size=4051419 2023-12-03T11:48:45.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.716+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=X2TP3TSM5HUL type=del 2023-12-03T11:48:45.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.717+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=0FQ4ZTUHQO0J type=del size=3994567 2023-12-03T11:48:45.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.718+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=YCKRBHBM9GU1 type=del 2023-12-03T11:48:45.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.719+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=U6FKP51ASM9N type=del size=1499581 2023-12-03T11:48:45.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.720+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~48XIIPQSB1N2.0 size=1048576 2023-12-03T11:48:45.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.721+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~48XIIPQSB1N2.1 size=767060 2023-12-03T11:48:45.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.723+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=48XIIPQSB1N2 tag=HYGITVW9ZI1Z type=write size=2864212 2023-12-03T11:48:45.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.724+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=QCERQHGWL3T4 type=del size=1096213 2023-12-03T11:48:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.725+0000 7f24574f5780 1 < canceled operation key=S0EH0Q0UA8HG tag=80BJW95SD415 type=del size=2376087 2023-12-03T11:48:45.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.726+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=A1007UMLFJM5 type=del size=3270454 2023-12-03T11:48:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.727+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~WERKX6QM2AP4.0 size=1048576 2023-12-03T11:48:45.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.728+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~WERKX6QM2AP4.1 size=429895 2023-12-03T11:48:45.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.731+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=WERKX6QM2AP4 tag=ELO3JTA6J0VG type=write size=2527047 2023-12-03T11:48:45.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.732+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=T9XKVNP8LDU0 type=del size=2977819 2023-12-03T11:48:45.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.734+0000 7f24574f5780 1 < suggested update operation key=35U7YCDYMX46 tag=JU64O7FQJC2W type=del size=1253625 2023-12-03T11:48:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.735+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=5PH3HML771I8 type=del size=4048492 2023-12-03T11:48:45.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.736+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=04MNM0VW8B6Q type=del size=657403 2023-12-03T11:48:45.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.738+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=0V6WCBJMQB7Z type=del size=2149271 2023-12-03T11:48:45.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.739+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~8I4Y72ZOQ9C5.0 size=1048576 2023-12-03T11:48:45.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.739+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~8I4Y72ZOQ9C5.1 size=45573 2023-12-03T11:48:45.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.742+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=8I4Y72ZOQ9C5 tag=4QMYSMQ5F6BU type=write size=2142725 2023-12-03T11:48:45.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.743+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=7WA8BTQOOJSN type=write size=579067 2023-12-03T11:48:45.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.744+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~Y97S9M9XUIV3.0 size=1048576 2023-12-03T11:48:45.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.745+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~Y97S9M9XUIV3.1 size=239995 2023-12-03T11:48:45.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.747+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=Y97S9M9XUIV3 tag=TZWK0UZHKHKZ type=write size=2337147 2023-12-03T11:48:45.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.749+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.0 size=1048576 2023-12-03T11:48:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.750+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.1 size=1048576 2023-12-03T11:48:45.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.751+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.2 size=240735 2023-12-03T11:48:45.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.751+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=JF0683ADHF80 tag=LJTKAP7P1703 type=write size=3386463 2023-12-03T11:48:45.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.752+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~IPD9JXCWQ3YM.0 size=795905 2023-12-03T11:48:45.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.754+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=IPD9JXCWQ3YM tag=TSOR1C9N8DPB type=write size=1844481 2023-12-03T11:48:45.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.756+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=CKHXF09XCBGC type=del size=3213344 2023-12-03T11:48:45.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.756+0000 7f24574f5780 1 < canceled part key=_multipart_863EOZDYRWH3.2~99W0P3OO163E.0 size=1048576 2023-12-03T11:48:45.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.757+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~99W0P3OO163E.1 size=897369 2023-12-03T11:48:45.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.759+0000 7f24574f5780 1 < canceled multipart upload key=863EOZDYRWH3 upload=99W0P3OO163E tag=XX635S81IP4I type=write size=2994521 2023-12-03T11:48:45.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.760+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=5IZSJNCCS5IR type=del size=3770925 2023-12-03T11:48:45.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.762+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.0 size=1048576 2023-12-03T11:48:45.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.763+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.1 size=1048576 2023-12-03T11:48:45.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.763+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~T7WSZNQA3SSZ.2 size=453332 2023-12-03T11:48:45.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.765+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=T7WSZNQA3SSZ tag=PGPTCPIYPQIW type=write size=3599060 2023-12-03T11:48:45.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.768+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~BSHZ3Z80YZ08.0 size=274166 2023-12-03T11:48:45.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.768+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=BSHZ3Z80YZ08 tag=BA0M1J8LQ78R type=write size=1322742 2023-12-03T11:48:45.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.769+0000 7f24574f5780 1 < canceled operation key=1QR0TE9DXU5Z tag=R82LP2GZFYMZ type=del size=966692 2023-12-03T11:48:45.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.770+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=WFSF65U20GCJ type=del size=3413393 2023-12-03T11:48:45.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.770+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~H0V7MHJ28YZ3.0 size=248878 2023-12-03T11:48:45.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.772+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=H0V7MHJ28YZ3 tag=VOIG5UP2US79 type=write size=1297454 2023-12-03T11:48:45.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.773+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=O44VB8ANP8UU type=del size=4192417 2023-12-03T11:48:45.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.775+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.0 size=1048576 2023-12-03T11:48:45.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.775+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.1 size=1048576 2023-12-03T11:48:45.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.776+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~QAAQTRK19Y0C.2 size=591890 2023-12-03T11:48:45.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.778+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=QAAQTRK19Y0C tag=Q9HVW00KFYOM type=write size=3737618 2023-12-03T11:48:45.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.780+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=JLWLTKL3HH7V type=del size=2900161 2023-12-03T11:48:45.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.781+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=RE0IRRM6J357 type=del 2023-12-03T11:48:45.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.782+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=9EA8LBLMUG9N type=del size=2257878 2023-12-03T11:48:45.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.783+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=0FQ4ZTUHQO0J type=del 2023-12-03T11:48:45.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.785+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=F5NUFG6JVS0Z type=write size=121343 2023-12-03T11:48:45.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.787+0000 7f24574f5780 1 < suggested remove operation key=S7Q5TCM499H1 tag=U6FKP51ASM9N type=del 2023-12-03T11:48:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.789+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~KPU6E7M1J7MM.0 size=1048576 2023-12-03T11:48:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.790+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~KPU6E7M1J7MM.1 size=544257 2023-12-03T11:48:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.790+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=KPU6E7M1J7MM tag=B32CRI42LIAV type=write size=2641409 2023-12-03T11:48:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.792+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=QCERQHGWL3T4 type=del 2023-12-03T11:48:45.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.794+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.0 size=1048576 2023-12-03T11:48:45.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.795+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.1 size=1048576 2023-12-03T11:48:45.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.796+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.2 size=884558 2023-12-03T11:48:45.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.796+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=LRHGD5XYLUD5 tag=O495C0C9X36D type=write size=4030286 2023-12-03T11:48:45.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.798+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=A1007UMLFJM5 type=del 2023-12-03T11:48:45.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.799+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=R9J7LO6WAR0B type=del size=1809790 2023-12-03T11:48:45.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.801+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=T9XKVNP8LDU0 type=del 2023-12-03T11:48:45.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.804+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.0 size=1048576 2023-12-03T11:48:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.805+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.1 size=1048576 2023-12-03T11:48:45.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.806+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.2 size=724445 2023-12-03T11:48:45.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.806+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=766B2L9A9ZS1 tag=C41XHNL1E0TW type=write size=3870173 2023-12-03T11:48:45.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.808+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=5PH3HML771I8 type=del 2023-12-03T11:48:45.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.809+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=LT9E7ZBZ56KR type=del size=1794178 2023-12-03T11:48:45.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.810+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=0V6WCBJMQB7Z type=del 2023-12-03T11:48:45.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.811+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=IHCQEJ1ST5A2 type=del size=3967504 2023-12-03T11:48:45.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.814+0000 7f24574f5780 1 < suggested update operation key=MH1SRAATMNXB tag=7WA8BTQOOJSN type=write size=579067 2023-12-03T11:48:45.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.815+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=0JZDO522F7EX type=del size=1030528 2023-12-03T11:48:45.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.816+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.0 size=1048576 2023-12-03T11:48:45.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.817+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.1 size=1048576 2023-12-03T11:48:45.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.818+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~JF0683ADHF80.2 size=240735 2023-12-03T11:48:45.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.820+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=JF0683ADHF80 tag=LJTKAP7P1703 type=write size=3386463 2023-12-03T11:48:45.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.822+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~WRK7HQNU4MXL.0 size=699083 2023-12-03T11:48:45.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.822+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=WRK7HQNU4MXL tag=8C4XC8WBBHSS type=write size=1747659 2023-12-03T11:48:45.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.823+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=CKHXF09XCBGC type=del 2023-12-03T11:48:45.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.825+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~DGP7N1KLNL3E.0 size=576748 2023-12-03T11:48:45.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.825+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=DGP7N1KLNL3E tag=I33SB9KL5SBP type=write size=1625324 2023-12-03T11:48:45.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.827+0000 7f24574f5780 1 < suggested update operation key=MY3K9M4QRS3L tag=5IZSJNCCS5IR type=del size=3770925 2023-12-03T11:48:45.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.829+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=JDFMVUCYJ0R8 type=del size=4061566 2023-12-03T11:48:45.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.830+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~BSHZ3Z80YZ08.0 size=274166 2023-12-03T11:48:45.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.831+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=BSHZ3Z80YZ08 tag=BA0M1J8LQ78R type=write size=1322742 2023-12-03T11:48:45.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.834+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.0 size=1048576 2023-12-03T11:48:45.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.834+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.1 size=1048576 2023-12-03T11:48:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.835+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.2 size=479643 2023-12-03T11:48:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.835+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=IUW36RCXTFID tag=0WRXP2K9PZLD type=write size=3625371 2023-12-03T11:48:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.837+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=WFSF65U20GCJ type=del 2023-12-03T11:48:45.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.839+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~HJRMTR43IPD1.0 size=1048576 2023-12-03T11:48:45.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.840+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~HJRMTR43IPD1.1 size=481287 2023-12-03T11:48:45.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.840+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=HJRMTR43IPD1 tag=5R3ULKU8Y04C type=write size=2578439 2023-12-03T11:48:45.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.841+0000 7f24574f5780 -1 < failed to complete operation key=MH1SRAATMNXB tag=O44VB8ANP8UU type=del size=4192417: (22) Invalid argument 2023-12-03T11:48:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.843+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~QP9K4CYO3LBL.0 size=212600 2023-12-03T11:48:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.843+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=QP9K4CYO3LBL tag=2S7020G7UODB type=write size=1261176 2023-12-03T11:48:45.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.844+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=JLWLTKL3HH7V type=del 2023-12-03T11:48:45.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.846+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.0 size=1048576 2023-12-03T11:48:45.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.847+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.1 size=1048576 2023-12-03T11:48:45.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.848+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.2 size=896390 2023-12-03T11:48:45.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.848+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=FJLMRG26UCIB tag=RI22P6PZCVGN type=write size=4042118 2023-12-03T11:48:45.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.849+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=9EA8LBLMUG9N type=del 2023-12-03T11:48:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.851+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=65BWL82YR1JM type=write size=479014 2023-12-03T11:48:45.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.852+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=F5NUFG6JVS0Z type=write size=121343 2023-12-03T11:48:45.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.857+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.0 size=1048576 2023-12-03T11:48:45.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.859+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.1 size=1048576 2023-12-03T11:48:45.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.861+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.2 size=1021538 2023-12-03T11:48:45.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.861+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=0VVDYSEV6HVV tag=7RJ8PRNYF75W type=write size=4167266 2023-12-03T11:48:45.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.861+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~KPU6E7M1J7MM.0 size=1048576 2023-12-03T11:48:45.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.862+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~KPU6E7M1J7MM.1 size=544257 2023-12-03T11:48:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.864+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=KPU6E7M1J7MM tag=B32CRI42LIAV type=write size=2641409 2023-12-03T11:48:45.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.865+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=9ZBPFTCNMP65 type=write size=175726 2023-12-03T11:48:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.866+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.0 size=1048576 2023-12-03T11:48:45.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.867+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.1 size=1048576 2023-12-03T11:48:45.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.868+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~LRHGD5XYLUD5.2 size=884558 2023-12-03T11:48:45.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.871+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=LRHGD5XYLUD5 tag=O495C0C9X36D type=write size=4030286 2023-12-03T11:48:45.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.872+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=0FHMDYA6QIXF type=del size=2140135 2023-12-03T11:48:45.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.873+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=R9J7LO6WAR0B type=del 2023-12-03T11:48:45.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.875+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~UO7OJVE4Q2NF.0 size=883197 2023-12-03T11:48:45.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.875+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=UO7OJVE4Q2NF tag=0EHUSCO95N5W type=write size=1931773 2023-12-03T11:48:45.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.876+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.0 size=1048576 2023-12-03T11:48:45.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.877+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.1 size=1048576 2023-12-03T11:48:45.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.878+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~766B2L9A9ZS1.2 size=724445 2023-12-03T11:48:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.880+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=766B2L9A9ZS1 tag=C41XHNL1E0TW type=write size=3870173 2023-12-03T11:48:45.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.882+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=U98ZT72ZD7OK type=del size=1048836 2023-12-03T11:48:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.883+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=LT9E7ZBZ56KR type=del 2023-12-03T11:48:45.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.884+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=F61P9VMRCPON type=del size=4121439 2023-12-03T11:48:45.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.885+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=IHCQEJ1ST5A2 type=del 2023-12-03T11:48:45.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.886+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=T9NNKJS0YEEL type=del size=3260231 2023-12-03T11:48:45.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.888+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=0JZDO522F7EX type=del 2023-12-03T11:48:45.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.889+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=S90ISTROCM9K type=write size=663581 2023-12-03T11:48:45.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.890+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~WRK7HQNU4MXL.0 size=699083 2023-12-03T11:48:45.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.892+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=WRK7HQNU4MXL tag=8C4XC8WBBHSS type=write size=1747659 2023-12-03T11:48:45.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.893+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=UJHENSBTZ47H type=del size=2998543 2023-12-03T11:48:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.894+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~DGP7N1KLNL3E.0 size=576748 2023-12-03T11:48:45.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.896+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=DGP7N1KLNL3E tag=I33SB9KL5SBP type=write size=1625324 2023-12-03T11:48:45.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.898+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=8LKVK794H27I type=write size=1025301 2023-12-03T11:48:45.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.899+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=JDFMVUCYJ0R8 type=del 2023-12-03T11:48:45.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.900+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=R2FIULTB8CAC type=write size=808597 2023-12-03T11:48:45.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.901+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.0 size=1048576 2023-12-03T11:48:45.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.901+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.1 size=1048576 2023-12-03T11:48:45.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.902+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IUW36RCXTFID.2 size=479643 2023-12-03T11:48:45.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.904+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=IUW36RCXTFID tag=0WRXP2K9PZLD type=write size=3625371 2023-12-03T11:48:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.905+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=LIQL97HM5FI2 type=del size=1802334 2023-12-03T11:48:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.906+0000 7f24574f5780 1 < canceled part key=_multipart_U043YAPGUBT5.2~HJRMTR43IPD1.0 size=1048576 2023-12-03T11:48:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.907+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~HJRMTR43IPD1.1 size=481287 2023-12-03T11:48:45.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.909+0000 7f24574f5780 1 < canceled multipart upload key=U043YAPGUBT5 upload=HJRMTR43IPD1 tag=5R3ULKU8Y04C type=write size=2578439 2023-12-03T11:48:45.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.910+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=DSJCMONUXF8K type=del size=2042603 2023-12-03T11:48:45.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.911+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~QP9K4CYO3LBL.0 size=212600 2023-12-03T11:48:45.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.914+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=QP9K4CYO3LBL tag=2S7020G7UODB type=write size=1261176 2023-12-03T11:48:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.916+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~G4YXET2N0N5K.0 size=1048576 2023-12-03T11:48:45.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.917+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~G4YXET2N0N5K.1 size=403746 2023-12-03T11:48:45.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.917+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=G4YXET2N0N5K tag=8OJW41UIX38V type=write size=2500898 2023-12-03T11:48:45.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.918+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.0 size=1048576 2023-12-03T11:48:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.919+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.1 size=1048576 2023-12-03T11:48:45.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.920+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~FJLMRG26UCIB.2 size=896390 2023-12-03T11:48:45.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.922+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=FJLMRG26UCIB tag=RI22P6PZCVGN type=write size=4042118 2023-12-03T11:48:45.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.923+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=5FIRECUH9S3I type=write size=654161 2023-12-03T11:48:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.925+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=65BWL82YR1JM type=write size=479014 2023-12-03T11:48:45.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.927+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~79HMYSETP8HQ.0 size=1048576 2023-12-03T11:48:45.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.928+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~79HMYSETP8HQ.1 size=440681 2023-12-03T11:48:45.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.928+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=79HMYSETP8HQ tag=AIHG0VI6HI97 type=write size=2537833 2023-12-03T11:48:45.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.929+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.0 size=1048576 2023-12-03T11:48:45.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.930+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.1 size=1048576 2023-12-03T11:48:45.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.931+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~0VVDYSEV6HVV.2 size=1021538 2023-12-03T11:48:45.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.933+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=0VVDYSEV6HVV tag=7RJ8PRNYF75W type=write size=4167266 2023-12-03T11:48:45.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.934+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=9IO391ZSWN43 type=del size=2512530 2023-12-03T11:48:45.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.937+0000 7f24574f5780 1 < suggested remove operation key=XZFHFS76WNR4 tag=9ZBPFTCNMP65 type=write 2023-12-03T11:48:45.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.938+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=FYVE9J6V10YQ type=del size=2692276 2023-12-03T11:48:45.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.940+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=0FHMDYA6QIXF type=del size=2140135 2023-12-03T11:48:45.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.943+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~F19YSXQKG04T.0 size=669965 2023-12-03T11:48:45.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.943+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=F19YSXQKG04T tag=ATZOEW9JGZS6 type=write size=1718541 2023-12-03T11:48:45.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.944+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~UO7OJVE4Q2NF.0 size=883197 2023-12-03T11:48:45.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.946+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=UO7OJVE4Q2NF tag=0EHUSCO95N5W type=write size=1931773 2023-12-03T11:48:45.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.948+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.0 size=1048576 2023-12-03T11:48:45.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.949+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.1 size=1048576 2023-12-03T11:48:45.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.950+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.2 size=764022 2023-12-03T11:48:45.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.950+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=HYLJL32KZDB0 tag=6H28QTKWX7OS type=write size=3909750 2023-12-03T11:48:45.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.953+0000 7f24574f5780 1 < suggested update operation key=WZMWCSAZX8RB tag=U98ZT72ZD7OK type=del size=1048836 2023-12-03T11:48:45.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.955+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.0 size=1048576 2023-12-03T11:48:45.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.956+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.1 size=1048576 2023-12-03T11:48:45.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.957+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.2 size=467935 2023-12-03T11:48:45.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.957+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=A9RCEVDE9DOO tag=4CE3HXGOCUR0 type=write size=3613663 2023-12-03T11:48:45.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.958+0000 7f24574f5780 1 < canceled operation key=35U7YCDYMX46 tag=F61P9VMRCPON type=del size=4121439 2023-12-03T11:48:45.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.959+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=R30NU1G3IPSU type=del size=4091388 2023-12-03T11:48:45.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.961+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=T9NNKJS0YEEL type=del 2023-12-03T11:48:45.964 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.963+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.0 size=1048576 2023-12-03T11:48:45.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.964+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.1 size=1048576 2023-12-03T11:48:45.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.965+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.2 size=190547 2023-12-03T11:48:45.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.965+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=WPMQJ8I907QU tag=NWT3X02HIPGX type=write size=3336275 2023-12-03T11:48:45.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.966+0000 7f24574f5780 1 < completed write operation key=MH1SRAATMNXB tag=S90ISTROCM9K type=write size=663581 2023-12-03T11:48:45.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.970+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.0 size=1048576 2023-12-03T11:48:45.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.971+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.1 size=1048576 2023-12-03T11:48:45.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.972+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.2 size=838408 2023-12-03T11:48:45.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.972+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=L26CK7NG7KDT tag=W9DETNBOA4ON type=write size=3984136 2023-12-03T11:48:45.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.973+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=UJHENSBTZ47H type=del 2023-12-03T11:48:45.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.974+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=9UZSJFSS01OT type=del size=2411588 2023-12-03T11:48:45.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.975+0000 7f24574f5780 1 < completed write operation key=QVLPCWC4T8N8 tag=8LKVK794H27I type=write size=1025301 2023-12-03T11:48:45.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.977+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~I0TALJTWJPOE.0 size=1048576 2023-12-03T11:48:45.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.978+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~I0TALJTWJPOE.1 size=547637 2023-12-03T11:48:45.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.978+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=I0TALJTWJPOE tag=WNW3WQPNOG8Q type=write size=2644789 2023-12-03T11:48:45.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.980+0000 7f24574f5780 1 < suggested remove operation key=S7Q5TCM499H1 tag=R2FIULTB8CAC type=write 2023-12-03T11:48:45.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.981+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=WEHI8YULVQF6 type=del size=2275683 2023-12-03T11:48:45.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.983+0000 7f24574f5780 1 < canceled operation key=LS4OEAGZWQXC tag=LIQL97HM5FI2 type=del size=1802334 2023-12-03T11:48:45.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.984+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=6GAHR6G54KJ2 type=del size=2648128 2023-12-03T11:48:45.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.985+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=DSJCMONUXF8K type=del 2023-12-03T11:48:45.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.987+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~9CRZ56XI7SJ3.0 size=176673 2023-12-03T11:48:45.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.987+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=9CRZ56XI7SJ3 tag=LPJBDXL9MD4I type=write size=1225249 2023-12-03T11:48:45.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.988+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~G4YXET2N0N5K.0 size=1048576 2023-12-03T11:48:45.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.989+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~G4YXET2N0N5K.1 size=403746 2023-12-03T11:48:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.991+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=G4YXET2N0N5K tag=8OJW41UIX38V type=write size=2500898 2023-12-03T11:48:45.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.993+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.0 size=1048576 2023-12-03T11:48:45.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.994+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.1 size=1048576 2023-12-03T11:48:45.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.995+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.2 size=687803 2023-12-03T11:48:45.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.995+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=C5801MY0I7HB tag=JW4484DK0KTI type=write size=3833531 2023-12-03T11:48:45.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.996+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=5FIRECUH9S3I type=write size=654161 2023-12-03T11:48:45.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.998+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~E600AQE3VP2G.0 size=1048576 2023-12-03T11:48:46.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.999+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~E600AQE3VP2G.1 size=127365 2023-12-03T11:48:46.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:45.999+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=E600AQE3VP2G tag=10FOO6FH9M8N type=write size=2224517 2023-12-03T11:48:46.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.000+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~79HMYSETP8HQ.0 size=1048576 2023-12-03T11:48:46.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.001+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~79HMYSETP8HQ.1 size=440681 2023-12-03T11:48:46.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.003+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=79HMYSETP8HQ tag=AIHG0VI6HI97 type=write size=2537833 2023-12-03T11:48:46.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.006+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~9F0C4F2S9V5N.0 size=931388 2023-12-03T11:48:46.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.006+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=9F0C4F2S9V5N tag=FQV1EYPGAP4G type=write size=1979964 2023-12-03T11:48:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.007+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=9IO391ZSWN43 type=del 2023-12-03T11:48:46.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.009+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~ZS7PRWQMXI5T.0 size=301610 2023-12-03T11:48:46.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.009+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=ZS7PRWQMXI5T tag=1IJTC1RCCMN5 type=write size=1350186 2023-12-03T11:48:46.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.011+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=FYVE9J6V10YQ type=del 2023-12-03T11:48:46.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.012+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=Y4KVU0HYC7HV type=del size=2302845 2023-12-03T11:48:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.013+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~F19YSXQKG04T.0 size=669965 2023-12-03T11:48:46.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.015+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=F19YSXQKG04T tag=ATZOEW9JGZS6 type=write size=1718541 2023-12-03T11:48:46.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.016+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=6AU8G4YPCJU6 type=del size=3734056 2023-12-03T11:48:46.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.017+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.0 size=1048576 2023-12-03T11:48:46.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.018+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.1 size=1048576 2023-12-03T11:48:46.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.019+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~HYLJL32KZDB0.2 size=764022 2023-12-03T11:48:46.022 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.021+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=HYLJL32KZDB0 tag=6H28QTKWX7OS type=write size=3909750 2023-12-03T11:48:46.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.022+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=23E1E90835BF type=del size=1308111 2023-12-03T11:48:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.023+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.0 size=1048576 2023-12-03T11:48:46.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.024+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.1 size=1048576 2023-12-03T11:48:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.025+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~A9RCEVDE9DOO.2 size=467935 2023-12-03T11:48:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.027+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=A9RCEVDE9DOO tag=4CE3HXGOCUR0 type=write size=3613663 2023-12-03T11:48:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.028+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=M5UZPCSOB5WQ type=del size=2901951 2023-12-03T11:48:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.030+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=R30NU1G3IPSU type=del 2023-12-03T11:48:46.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.031+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=Z2L92LOTGX4Z type=del size=1454986 2023-12-03T11:48:46.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.032+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.0 size=1048576 2023-12-03T11:48:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.033+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.1 size=1048576 2023-12-03T11:48:46.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.034+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~WPMQJ8I907QU.2 size=190547 2023-12-03T11:48:46.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.036+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=WPMQJ8I907QU tag=NWT3X02HIPGX type=write size=3336275 2023-12-03T11:48:46.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.038+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~SNASQD8EJ6R5.0 size=1048576 2023-12-03T11:48:46.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.039+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~SNASQD8EJ6R5.1 size=142317 2023-12-03T11:48:46.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.039+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=SNASQD8EJ6R5 tag=I7LAHBZCR9K0 type=write size=2239469 2023-12-03T11:48:46.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.040+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.0 size=1048576 2023-12-03T11:48:46.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.041+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.1 size=1048576 2023-12-03T11:48:46.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.042+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~L26CK7NG7KDT.2 size=838408 2023-12-03T11:48:46.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.044+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=L26CK7NG7KDT tag=W9DETNBOA4ON type=write size=3984136 2023-12-03T11:48:46.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.045+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=Z7OH9EDOA4LC type=write size=139655 2023-12-03T11:48:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.046+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=9UZSJFSS01OT type=del 2023-12-03T11:48:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.048+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~GBR2M7HA3CM6.0 size=902354 2023-12-03T11:48:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.048+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=GBR2M7HA3CM6 tag=67DJ2LT2RUZT type=write size=1950930 2023-12-03T11:48:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.049+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~I0TALJTWJPOE.0 size=1048576 2023-12-03T11:48:46.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.050+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~I0TALJTWJPOE.1 size=547637 2023-12-03T11:48:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.052+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=I0TALJTWJPOE tag=WNW3WQPNOG8Q type=write size=2644789 2023-12-03T11:48:46.054 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.053+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=Y5A9Y5RU2M5P type=write size=180880 2023-12-03T11:48:46.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.054+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=WEHI8YULVQF6 type=del 2023-12-03T11:48:46.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.055+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=375NFNU2RCPW type=del size=1229600 2023-12-03T11:48:46.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.057+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=6GAHR6G54KJ2 type=del 2023-12-03T11:48:46.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.059+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~MQ6NIIYF17KJ.0 size=717523 2023-12-03T11:48:46.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.059+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=MQ6NIIYF17KJ tag=PNB2B719FCOS type=write size=1766099 2023-12-03T11:48:46.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.060+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~9CRZ56XI7SJ3.0 size=176673 2023-12-03T11:48:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.062+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=9CRZ56XI7SJ3 tag=LPJBDXL9MD4I type=write size=1225249 2023-12-03T11:48:46.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.064+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.0 size=1048576 2023-12-03T11:48:46.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.065+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.1 size=1048576 2023-12-03T11:48:46.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.066+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.2 size=645467 2023-12-03T11:48:46.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.066+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=925CKZVZZ34Z tag=0ISGFX756S8N type=write size=3791195 2023-12-03T11:48:46.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.066+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.0 size=1048576 2023-12-03T11:48:46.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.067+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.1 size=1048576 2023-12-03T11:48:46.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.068+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~C5801MY0I7HB.2 size=687803 2023-12-03T11:48:46.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.070+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=C5801MY0I7HB tag=JW4484DK0KTI type=write size=3833531 2023-12-03T11:48:46.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.072+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=S6TXOLI4Q71O type=del size=2751967 2023-12-03T11:48:46.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.073+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~E600AQE3VP2G.0 size=1048576 2023-12-03T11:48:46.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.074+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~E600AQE3VP2G.1 size=127365 2023-12-03T11:48:46.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.076+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=E600AQE3VP2G tag=10FOO6FH9M8N type=write size=2224517 2023-12-03T11:48:46.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.078+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=ZBGNTM5TT7TV type=del size=1078117 2023-12-03T11:48:46.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.079+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~9F0C4F2S9V5N.0 size=931388 2023-12-03T11:48:46.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.081+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=9F0C4F2S9V5N tag=FQV1EYPGAP4G type=write size=1979964 2023-12-03T11:48:46.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.082+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=QLTGNLD8ECSL type=del size=3356555 2023-12-03T11:48:46.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.083+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~ZS7PRWQMXI5T.0 size=301610 2023-12-03T11:48:46.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.085+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=ZS7PRWQMXI5T tag=1IJTC1RCCMN5 type=write size=1350186 2023-12-03T11:48:46.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.087+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=20R737E6D1H9 type=del size=3045748 2023-12-03T11:48:46.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.088+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=Y4KVU0HYC7HV type=del 2023-12-03T11:48:46.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.089+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=24KDE32E32TV type=del size=3747087 2023-12-03T11:48:46.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.091+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=6AU8G4YPCJU6 type=del 2023-12-03T11:48:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.092+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=HYWT4P3BT84V type=write size=744882 2023-12-03T11:48:46.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.093+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=23E1E90835BF type=del 2023-12-03T11:48:46.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.095+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=XCBG6XT7BYIU type=del size=315144 2023-12-03T11:48:46.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.096+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=M5UZPCSOB5WQ type=del 2023-12-03T11:48:46.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.099+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~5V0EHOM468PF.0 size=1048576 2023-12-03T11:48:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.100+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~5V0EHOM468PF.1 size=358847 2023-12-03T11:48:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.100+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=5V0EHOM468PF tag=2BX48FHD22JS type=write size=2455999 2023-12-03T11:48:46.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.102+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=Z2L92LOTGX4Z type=del 2023-12-03T11:48:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.103+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=4MLELK4EJ6X0 type=write size=474716 2023-12-03T11:48:46.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.105+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~SNASQD8EJ6R5.0 size=1048576 2023-12-03T11:48:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.106+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~SNASQD8EJ6R5.1 size=142317 2023-12-03T11:48:46.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.108+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=SNASQD8EJ6R5 tag=I7LAHBZCR9K0 type=write size=2239469 2023-12-03T11:48:46.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.110+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~OVXBXP5V5PN0.0 size=292340 2023-12-03T11:48:46.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.110+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=OVXBXP5V5PN0 tag=ESLW3YRKLD30 type=write size=1340916 2023-12-03T11:48:46.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.111+0000 7f24574f5780 1 < completed write operation key=XO42YZR60KQE tag=Z7OH9EDOA4LC type=write size=139655 2023-12-03T11:48:46.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.113+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.0 size=1048576 2023-12-03T11:48:46.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.114+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.1 size=1048576 2023-12-03T11:48:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.115+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.2 size=962738 2023-12-03T11:48:46.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.115+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=9PWNHUHNBZKW tag=5L4CD6HW762U type=write size=4108466 2023-12-03T11:48:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.116+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~GBR2M7HA3CM6.0 size=902354 2023-12-03T11:48:46.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.119+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=GBR2M7HA3CM6 tag=67DJ2LT2RUZT type=write size=1950930 2023-12-03T11:48:46.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.121+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~35FIX862G5WH.0 size=1048576 2023-12-03T11:48:46.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.122+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~35FIX862G5WH.1 size=706932 2023-12-03T11:48:46.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.122+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=35FIX862G5WH tag=S6AZ04PXWZR3 type=write size=2804084 2023-12-03T11:48:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.124+0000 7f24574f5780 1 < suggested remove operation key=WZMWCSAZX8RB tag=Y5A9Y5RU2M5P type=write 2023-12-03T11:48:46.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.125+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=CQJTG3355AVS type=del size=1392823 2023-12-03T11:48:46.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.128+0000 7f24574f5780 1 < suggested remove operation key=LS4OEAGZWQXC tag=375NFNU2RCPW type=del 2023-12-03T11:48:46.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.130+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~HLMI8EX25XZU.0 size=953728 2023-12-03T11:48:46.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.130+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=HLMI8EX25XZU tag=9RC9NPUU0MGB type=write size=2002304 2023-12-03T11:48:46.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.131+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~MQ6NIIYF17KJ.0 size=717523 2023-12-03T11:48:46.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.134+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=MQ6NIIYF17KJ tag=PNB2B719FCOS type=write size=1766099 2023-12-03T11:48:46.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.135+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=6DE2J9MFZX5T type=del size=3825646 2023-12-03T11:48:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.136+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.0 size=1048576 2023-12-03T11:48:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.137+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.1 size=1048576 2023-12-03T11:48:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.138+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~925CKZVZZ34Z.2 size=645467 2023-12-03T11:48:46.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.140+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=925CKZVZZ34Z tag=0ISGFX756S8N type=write size=3791195 2023-12-03T11:48:46.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.142+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.0 size=1048576 2023-12-03T11:48:46.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.143+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.1 size=1048576 2023-12-03T11:48:46.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.144+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.2 size=620927 2023-12-03T11:48:46.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.144+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=QT4BAGY70WLQ tag=5HBUT7PWP7HX type=write size=3766655 2023-12-03T11:48:46.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.146+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=S6TXOLI4Q71O type=del 2023-12-03T11:48:46.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.147+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=7Y7QTL4JSJUW type=del size=1065957 2023-12-03T11:48:46.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.148+0000 7f24574f5780 1 < canceled operation key=6WY91DFIRUN8 tag=ZBGNTM5TT7TV type=del size=1078117 2023-12-03T11:48:46.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.149+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=HTV9VQF3410E type=write size=691638 2023-12-03T11:48:46.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.150+0000 7f24574f5780 1 < canceled operation key=HM9JHOV8W24F tag=QLTGNLD8ECSL type=del size=3356555 2023-12-03T11:48:46.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.152+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=PNWYDYJYTDUI type=del size=1999115 2023-12-03T11:48:46.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.153+0000 7f24574f5780 1 < canceled operation key=1V39QA10M1HY tag=20R737E6D1H9 type=del size=3045748 2023-12-03T11:48:46.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.154+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=WK7LHXE6VWHR type=del size=2885818 2023-12-03T11:48:46.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.156+0000 7f24574f5780 1 < suggested update operation key=QVLPCWC4T8N8 tag=24KDE32E32TV type=del size=3747087 2023-12-03T11:48:46.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.158+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=3AT1FG1SV7Z7 type=del size=468826 2023-12-03T11:48:46.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.159+0000 7f24574f5780 1 < completed write operation key=6WY91DFIRUN8 tag=HYWT4P3BT84V type=write size=744882 2023-12-03T11:48:46.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.160+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=EAA4SUEHGS82 type=del size=592750 2023-12-03T11:48:46.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.161+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=XCBG6XT7BYIU type=del 2023-12-03T11:48:46.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.163+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~KBFYUNN39AP5.0 size=1048576 2023-12-03T11:48:46.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.164+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~KBFYUNN39AP5.1 size=121270 2023-12-03T11:48:46.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.164+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=KBFYUNN39AP5 tag=KEXFZI6MIFXX type=write size=2218422 2023-12-03T11:48:46.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.165+0000 7f24574f5780 1 < canceled part key=_multipart_PGTGVLK4DBN0.2~5V0EHOM468PF.0 size=1048576 2023-12-03T11:48:46.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.166+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~5V0EHOM468PF.1 size=358847 2023-12-03T11:48:46.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.168+0000 7f24574f5780 1 < canceled multipart upload key=PGTGVLK4DBN0 upload=5V0EHOM468PF tag=2BX48FHD22JS type=write size=2455999 2023-12-03T11:48:46.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.170+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~X2CAAY9GY53F.0 size=1048576 2023-12-03T11:48:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.171+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~X2CAAY9GY53F.1 size=468132 2023-12-03T11:48:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.171+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=X2CAAY9GY53F tag=CPET8BXLG5GT type=write size=2565284 2023-12-03T11:48:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.171+0000 7f24574f5780 -1 < no bi entry to suggest for operation key=LS4OEAGZWQXC tag=4MLELK4EJ6X0 type=write size=474716: (2) No such file or directory 2023-12-03T11:48:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.173+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~0WIAXKXWOBYY.0 size=757827 2023-12-03T11:48:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.173+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=0WIAXKXWOBYY tag=HBES1O8S1PYB type=write size=1806403 2023-12-03T11:48:46.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.175+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~OVXBXP5V5PN0.0 size=292340 2023-12-03T11:48:46.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.177+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=OVXBXP5V5PN0 tag=ESLW3YRKLD30 type=write size=1340916 2023-12-03T11:48:46.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.179+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.0 size=1048576 2023-12-03T11:48:46.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.181+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.1 size=1048576 2023-12-03T11:48:46.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.182+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.2 size=622393 2023-12-03T11:48:46.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.182+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=KRACG8SM7B7K tag=X4VVJAL1ZRZU type=write size=3768121 2023-12-03T11:48:46.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.183+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.0 size=1048576 2023-12-03T11:48:46.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.184+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.1 size=1048576 2023-12-03T11:48:46.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.185+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~9PWNHUHNBZKW.2 size=962738 2023-12-03T11:48:46.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.187+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=9PWNHUHNBZKW tag=5L4CD6HW762U type=write size=4108466 2023-12-03T11:48:46.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.190+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.0 size=1048576 2023-12-03T11:48:46.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.190+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.1 size=1048576 2023-12-03T11:48:46.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.193+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.2 size=1011622 2023-12-03T11:48:46.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.193+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=ZAB636VMH31V tag=BGA7YGDVYVGM type=write size=4157350 2023-12-03T11:48:46.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.194+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~35FIX862G5WH.0 size=1048576 2023-12-03T11:48:46.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.195+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~35FIX862G5WH.1 size=706932 2023-12-03T11:48:46.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.197+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=35FIX862G5WH tag=S6AZ04PXWZR3 type=write size=2804084 2023-12-03T11:48:46.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.199+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~IVEZZXDGOXH8.0 size=388075 2023-12-03T11:48:46.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.199+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=IVEZZXDGOXH8 tag=JRV9PP2ZZTPY type=write size=1436651 2023-12-03T11:48:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.200+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=CQJTG3355AVS type=del 2023-12-03T11:48:46.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.201+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=JMWGXL4JVUQP type=del size=1351011 2023-12-03T11:48:46.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.202+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~HLMI8EX25XZU.0 size=953728 2023-12-03T11:48:46.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.204+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=HLMI8EX25XZU tag=9RC9NPUU0MGB type=write size=2002304 2023-12-03T11:48:46.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.206+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.0 size=1048576 2023-12-03T11:48:46.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.207+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.1 size=1048576 2023-12-03T11:48:46.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.207+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.2 size=326413 2023-12-03T11:48:46.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.207+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=UJ233RSAPK14 tag=CBKZDOG58CO8 type=write size=3472141 2023-12-03T11:48:46.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.209+0000 7f24574f5780 1 < canceled operation key=BV8C0VSJ1GTW tag=6DE2J9MFZX5T type=del size=3825646 2023-12-03T11:48:46.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.210+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=8ZRTMYYT8DZE type=write size=18886 2023-12-03T11:48:46.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.211+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.0 size=1048576 2023-12-03T11:48:46.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.211+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.1 size=1048576 2023-12-03T11:48:46.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.212+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~QT4BAGY70WLQ.2 size=620927 2023-12-03T11:48:46.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.214+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=QT4BAGY70WLQ tag=5HBUT7PWP7HX type=write size=3766655 2023-12-03T11:48:46.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.216+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~1Q7MXRJ67JAQ.0 size=1048576 2023-12-03T11:48:46.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.217+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~1Q7MXRJ67JAQ.1 size=635879 2023-12-03T11:48:46.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.217+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=1Q7MXRJ67JAQ tag=72CHHBTCH19G type=write size=2733031 2023-12-03T11:48:46.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.219+0000 7f24574f5780 1 < suggested update operation key=S7Q5TCM499H1 tag=7Y7QTL4JSJUW type=del size=1065957 2023-12-03T11:48:46.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.222+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~KOH0EI1IJ8B9.0 size=926051 2023-12-03T11:48:46.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.222+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=KOH0EI1IJ8B9 tag=P0F3471LZYR4 type=write size=1974627 2023-12-03T11:48:46.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.223+0000 7f24574f5780 1 < canceled operation key=RABUR02D2ZMD tag=HTV9VQF3410E type=write size=691638 2023-12-03T11:48:46.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.225+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~N02PVVUXEAZX.0 size=1048576 2023-12-03T11:48:46.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.226+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~N02PVVUXEAZX.1 size=992488 2023-12-03T11:48:46.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.226+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=N02PVVUXEAZX tag=M83PV0VFD4CG type=write size=3089640 2023-12-03T11:48:46.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.227+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=PNWYDYJYTDUI type=del 2023-12-03T11:48:46.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.228+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=H0305HP571GD type=del size=127778 2023-12-03T11:48:46.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.230+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=WK7LHXE6VWHR type=del size=2885818 2023-12-03T11:48:46.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.232+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.0 size=1048576 2023-12-03T11:48:46.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.233+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.1 size=1048576 2023-12-03T11:48:46.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.234+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.2 size=648947 2023-12-03T11:48:46.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.234+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=YERN9JWL6JZU tag=MFNXO7Q1FCR6 type=write size=3794675 2023-12-03T11:48:46.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.235+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=3AT1FG1SV7Z7 type=del 2023-12-03T11:48:46.237 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.236+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=SWZ2H8ADFPVL type=del size=2370282 2023-12-03T11:48:46.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.237+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=EAA4SUEHGS82 type=del 2023-12-03T11:48:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.239+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=8SUWMIUDMLB0 type=del size=348277 2023-12-03T11:48:46.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.240+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~KBFYUNN39AP5.0 size=1048576 2023-12-03T11:48:46.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.242+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~KBFYUNN39AP5.1 size=121270 2023-12-03T11:48:46.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.244+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=KBFYUNN39AP5 tag=KEXFZI6MIFXX type=write size=2218422 2023-12-03T11:48:46.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.245+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=CVD923AY6APF type=write size=201022 2023-12-03T11:48:46.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.246+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~X2CAAY9GY53F.0 size=1048576 2023-12-03T11:48:46.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.247+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~X2CAAY9GY53F.1 size=468132 2023-12-03T11:48:46.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.249+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=X2CAAY9GY53F tag=CPET8BXLG5GT type=write size=2565284 2023-12-03T11:48:46.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.250+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=OFOEZQRNFHMH type=del size=3626567 2023-12-03T11:48:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.251+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~0WIAXKXWOBYY.0 size=757827 2023-12-03T11:48:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.253+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=0WIAXKXWOBYY tag=HBES1O8S1PYB type=write size=1806403 2023-12-03T11:48:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.254+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=N5X3VE9G3SPP type=del size=2512779 2023-12-03T11:48:46.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.256+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.0 size=1048576 2023-12-03T11:48:46.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.257+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.1 size=1048576 2023-12-03T11:48:46.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.257+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~KRACG8SM7B7K.2 size=622393 2023-12-03T11:48:46.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.259+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=KRACG8SM7B7K tag=X4VVJAL1ZRZU type=write size=3768121 2023-12-03T11:48:46.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.261+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~1TYROHX8X5EZ.0 size=537319 2023-12-03T11:48:46.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.261+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=1TYROHX8X5EZ tag=KFRJ0LKX9DF0 type=write size=1585895 2023-12-03T11:48:46.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.263+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.0 size=1048576 2023-12-03T11:48:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.264+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.1 size=1048576 2023-12-03T11:48:46.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.265+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~ZAB636VMH31V.2 size=1011622 2023-12-03T11:48:46.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.267+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=ZAB636VMH31V tag=BGA7YGDVYVGM type=write size=4157350 2023-12-03T11:48:46.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.268+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=56AH6BWLXWMZ type=del size=3827709 2023-12-03T11:48:46.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.268+0000 7f24574f5780 1 < canceled part key=_multipart_XZFHFS76WNR4.2~IVEZZXDGOXH8.0 size=388075 2023-12-03T11:48:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.270+0000 7f24574f5780 1 < canceled multipart upload key=XZFHFS76WNR4 upload=IVEZZXDGOXH8 tag=JRV9PP2ZZTPY type=write size=1436651 2023-12-03T11:48:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.271+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=V1NECIPP5DKX type=del size=2352821 2023-12-03T11:48:46.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.272+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=JMWGXL4JVUQP type=del 2023-12-03T11:48:46.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.275+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~77V1EPCORJUR.0 size=289230 2023-12-03T11:48:46.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.275+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=77V1EPCORJUR tag=8W0FI38B3FJ6 type=write size=1337806 2023-12-03T11:48:46.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.276+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.0 size=1048576 2023-12-03T11:48:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.277+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.1 size=1048576 2023-12-03T11:48:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.278+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~UJ233RSAPK14.2 size=326413 2023-12-03T11:48:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.280+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=UJ233RSAPK14 tag=CBKZDOG58CO8 type=write size=3472141 2023-12-03T11:48:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.282+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~HROABJ6UFUPC.0 size=744322 2023-12-03T11:48:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.282+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=HROABJ6UFUPC tag=CBXO826C2WUB type=write size=1792898 2023-12-03T11:48:46.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.284+0000 7f24574f5780 1 < suggested remove operation key=WD7U31FW3NGZ tag=8ZRTMYYT8DZE type=write 2023-12-03T11:48:46.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.285+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=VGNPXI4JBE2A type=del size=3182703 2023-12-03T11:48:46.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.286+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~1Q7MXRJ67JAQ.0 size=1048576 2023-12-03T11:48:46.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.287+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~1Q7MXRJ67JAQ.1 size=635879 2023-12-03T11:48:46.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.289+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=1Q7MXRJ67JAQ tag=72CHHBTCH19G type=write size=2733031 2023-12-03T11:48:46.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.290+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=XB6CBUKA5PTH type=write size=693976 2023-12-03T11:48:46.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.292+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~KOH0EI1IJ8B9.0 size=926051 2023-12-03T11:48:46.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.294+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=KOH0EI1IJ8B9 tag=P0F3471LZYR4 type=write size=1974627 2023-12-03T11:48:46.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.296+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~L1AON4W67QXO.0 size=1048576 2023-12-03T11:48:46.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.298+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~L1AON4W67QXO.1 size=426721 2023-12-03T11:48:46.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.298+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=L1AON4W67QXO tag=UE9RZCCB96KM type=write size=2523873 2023-12-03T11:48:46.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.299+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~N02PVVUXEAZX.0 size=1048576 2023-12-03T11:48:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.300+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~N02PVVUXEAZX.1 size=992488 2023-12-03T11:48:46.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.303+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=N02PVVUXEAZX tag=M83PV0VFD4CG type=write size=3089640 2023-12-03T11:48:46.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.305+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.0 size=1048576 2023-12-03T11:48:46.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.306+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.1 size=1048576 2023-12-03T11:48:46.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.307+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.2 size=2414 2023-12-03T11:48:46.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.307+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=6BGQUT8ZO82T tag=6KC2M1TW8954 type=write size=3148142 2023-12-03T11:48:46.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.309+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=H0305HP571GD type=del 2023-12-03T11:48:46.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.311+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=KGMFK8N1A3KH type=del size=1142273 2023-12-03T11:48:46.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.312+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.0 size=1048576 2023-12-03T11:48:46.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.313+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.1 size=1048576 2023-12-03T11:48:46.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.314+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~YERN9JWL6JZU.2 size=648947 2023-12-03T11:48:46.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.316+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=YERN9JWL6JZU tag=MFNXO7Q1FCR6 type=write size=3794675 2023-12-03T11:48:46.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.317+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=YLFV6L8YUUE4 type=del size=3883528 2023-12-03T11:48:46.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.320+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=SWZ2H8ADFPVL type=del 2023-12-03T11:48:46.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.321+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=13X6VAXQP6TY type=del size=1597344 2023-12-03T11:48:46.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.323+0000 7f24574f5780 1 < suggested remove operation key=WZMWCSAZX8RB tag=8SUWMIUDMLB0 type=del 2023-12-03T11:48:46.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.325+0000 7f24574f5780 1 > prepared part key=_multipart_RABUR02D2ZMD.2~XLIBP4LQGD05.0 size=271556 2023-12-03T11:48:46.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.325+0000 7f24574f5780 1 > prepared multipart upload key=RABUR02D2ZMD upload=XLIBP4LQGD05 tag=M0OOLM8SJ383 type=write size=1320132 2023-12-03T11:48:46.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.326+0000 7f24574f5780 1 < completed write operation key=XO42YZR60KQE tag=CVD923AY6APF type=write size=201022 2023-12-03T11:48:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.328+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.0 size=1048576 2023-12-03T11:48:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.331+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.1 size=1048576 2023-12-03T11:48:46.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.332+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.2 size=851458 2023-12-03T11:48:46.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.332+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=BT95HH88T67H tag=42XU9PQGLLTZ type=write size=3997186 2023-12-03T11:48:46.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.333+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=OFOEZQRNFHMH type=del 2023-12-03T11:48:46.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.334+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=EJGN44FIO7Z3 type=del size=2284782 2023-12-03T11:48:46.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.335+0000 7f24574f5780 -1 < failed to complete operation key=WD7U31FW3NGZ tag=N5X3VE9G3SPP type=del size=2512779: (22) Invalid argument 2023-12-03T11:48:46.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.336+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=X1CUMN7BIKRG type=del size=3240898 2023-12-03T11:48:46.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.338+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~1TYROHX8X5EZ.0 size=537319 2023-12-03T11:48:46.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.340+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=1TYROHX8X5EZ tag=KFRJ0LKX9DF0 type=write size=1585895 2023-12-03T11:48:46.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.341+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=1GRXN0CIZVXY type=del size=2781886 2023-12-03T11:48:46.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.342+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=56AH6BWLXWMZ type=del 2023-12-03T11:48:46.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.343+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=L4RYKUNEX5YG type=del size=3691278 2023-12-03T11:48:46.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.344+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=V1NECIPP5DKX type=del 2023-12-03T11:48:46.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.345+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=7ODLA5OTDCTH type=del size=1169467 2023-12-03T11:48:46.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.346+0000 7f24574f5780 1 < canceled part key=_multipart_HM9JHOV8W24F.2~77V1EPCORJUR.0 size=289230 2023-12-03T11:48:46.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.347+0000 7f24574f5780 1 < canceled multipart upload key=HM9JHOV8W24F upload=77V1EPCORJUR tag=8W0FI38B3FJ6 type=write size=1337806 2023-12-03T11:48:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.350+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~LVB2LE0QP83J.0 size=1048576 2023-12-03T11:48:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.350+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~LVB2LE0QP83J.1 size=847627 2023-12-03T11:48:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.350+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=LVB2LE0QP83J tag=JVAA73P1LLWL type=write size=2944779 2023-12-03T11:48:46.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.352+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~HROABJ6UFUPC.0 size=744322 2023-12-03T11:48:46.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.354+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=HROABJ6UFUPC tag=CBXO826C2WUB type=write size=1792898 2023-12-03T11:48:46.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.355+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=M3RT05FXHDE5 type=write size=838761 2023-12-03T11:48:46.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.356+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=VGNPXI4JBE2A type=del 2023-12-03T11:48:46.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.357+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=5K7D63Q9L6TQ type=del size=2974917 2023-12-03T11:48:46.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.358+0000 7f24574f5780 1 < completed write operation key=1V1C59DP8YPF tag=XB6CBUKA5PTH type=write size=693976 2023-12-03T11:48:46.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.359+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=KZS870F4WJEM type=del size=913639 2023-12-03T11:48:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.361+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~L1AON4W67QXO.0 size=1048576 2023-12-03T11:48:46.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.362+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~L1AON4W67QXO.1 size=426721 2023-12-03T11:48:46.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.364+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=L1AON4W67QXO tag=UE9RZCCB96KM type=write size=2523873 2023-12-03T11:48:46.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.365+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=PXZG24LL8HUW type=del size=2486014 2023-12-03T11:48:46.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.366+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.0 size=1048576 2023-12-03T11:48:46.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.367+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.1 size=1048576 2023-12-03T11:48:46.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.368+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~6BGQUT8ZO82T.2 size=2414 2023-12-03T11:48:46.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.370+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=6BGQUT8ZO82T tag=6KC2M1TW8954 type=write size=3148142 2023-12-03T11:48:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.372+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~I56WB6ZZQHAH.0 size=1026434 2023-12-03T11:48:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.372+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=I56WB6ZZQHAH tag=QK4QVGF3D625 type=write size=2075010 2023-12-03T11:48:46.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.373+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=KGMFK8N1A3KH type=del 2023-12-03T11:48:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.375+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~85B1966F8FZI.0 size=1048576 2023-12-03T11:48:46.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.376+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~85B1966F8FZI.1 size=958048 2023-12-03T11:48:46.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.376+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=85B1966F8FZI tag=WV9F8F67Q7FR type=write size=3055200 2023-12-03T11:48:46.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.377+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=YLFV6L8YUUE4 type=del 2023-12-03T11:48:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.380+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~72B8DPJZ00KY.0 size=1048576 2023-12-03T11:48:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.381+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~72B8DPJZ00KY.1 size=107282 2023-12-03T11:48:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.381+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=72B8DPJZ00KY tag=PRSZID8B7IX4 type=write size=2204434 2023-12-03T11:48:46.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.383+0000 7f24574f5780 1 < suggested update operation key=1V39QA10M1HY tag=13X6VAXQP6TY type=del size=1597344 2023-12-03T11:48:46.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.385+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~U70KD52WH1DF.0 size=314453 2023-12-03T11:48:46.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.385+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=U70KD52WH1DF tag=WBF259VINXES type=write size=1363029 2023-12-03T11:48:46.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.386+0000 7f24574f5780 1 < completed part key=_multipart_RABUR02D2ZMD.2~XLIBP4LQGD05.0 size=271556 2023-12-03T11:48:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.390+0000 7f24574f5780 1 < completed multipart upload key=RABUR02D2ZMD upload=XLIBP4LQGD05 tag=M0OOLM8SJ383 type=write size=1320132 2023-12-03T11:48:46.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.392+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.0 size=1048576 2023-12-03T11:48:46.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.393+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.1 size=1048576 2023-12-03T11:48:46.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.394+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.2 size=370529 2023-12-03T11:48:46.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.394+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=VI5Q7EOG7TYS tag=092DQ1LAJT93 type=write size=3516257 2023-12-03T11:48:46.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.395+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.0 size=1048576 2023-12-03T11:48:46.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.395+0000 7f24574f5780 1 < canceled part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.1 size=1048576 2023-12-03T11:48:46.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.396+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~BT95HH88T67H.2 size=851458 2023-12-03T11:48:46.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.398+0000 7f24574f5780 1 < canceled multipart upload key=U043YAPGUBT5 upload=BT95HH88T67H tag=42XU9PQGLLTZ type=write size=3997186 2023-12-03T11:48:46.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.401+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~LLVO6URUW41X.0 size=1048576 2023-12-03T11:48:46.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.402+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~LLVO6URUW41X.1 size=196584 2023-12-03T11:48:46.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.402+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=LLVO6URUW41X tag=L6C04S3G9SDF type=write size=2293736 2023-12-03T11:48:46.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.404+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=EJGN44FIO7Z3 type=del 2023-12-03T11:48:46.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.406+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~KLGWXBCALUJE.0 size=1048576 2023-12-03T11:48:46.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.407+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~KLGWXBCALUJE.1 size=300434 2023-12-03T11:48:46.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.407+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=KLGWXBCALUJE tag=NMQT428L0XRH type=write size=2397586 2023-12-03T11:48:46.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.408+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=X1CUMN7BIKRG type=del 2023-12-03T11:48:46.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.409+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=HPECYIAWBFLT type=del size=1176811 2023-12-03T11:48:46.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.410+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=1GRXN0CIZVXY type=del 2023-12-03T11:48:46.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.412+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=WLHSNKU98XG3 type=del size=3277452 2023-12-03T11:48:46.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.413+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=L4RYKUNEX5YG type=del 2023-12-03T11:48:46.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.414+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=5Y8VZYUE5PH4 type=del size=73569 2023-12-03T11:48:46.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.418+0000 7f24574f5780 1 < suggested remove operation key=3EOEF9UMATIE tag=7ODLA5OTDCTH type=del 2023-12-03T11:48:46.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.419+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=UIM1UJ087D01 type=del size=1033268 2023-12-03T11:48:46.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.421+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~LVB2LE0QP83J.0 size=1048576 2023-12-03T11:48:46.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.421+0000 7f24574f5780 1 < canceled part key=_multipart_S73M6UFMX5NH.2~LVB2LE0QP83J.1 size=847627 2023-12-03T11:48:46.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.423+0000 7f24574f5780 1 < canceled multipart upload key=S73M6UFMX5NH upload=LVB2LE0QP83J tag=JVAA73P1LLWL type=write size=2944779 2023-12-03T11:48:46.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.425+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.0 size=1048576 2023-12-03T11:48:46.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.426+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.1 size=1048576 2023-12-03T11:48:46.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.427+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.2 size=583957 2023-12-03T11:48:46.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.427+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=AW02QC3WVHXJ tag=9Y045NL5JIYW type=write size=3729685 2023-12-03T11:48:46.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.428+0000 7f24574f5780 1 < canceled operation key=ON2E6H35N9FJ tag=M3RT05FXHDE5 type=write size=838761 2023-12-03T11:48:46.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.431+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~RBBP3BIERWWW.0 size=1048576 2023-12-03T11:48:46.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.432+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~RBBP3BIERWWW.1 size=650567 2023-12-03T11:48:46.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.432+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=RBBP3BIERWWW tag=81E7TW201S3X type=write size=2747719 2023-12-03T11:48:46.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.433+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=5K7D63Q9L6TQ type=del 2023-12-03T11:48:46.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.435+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~2EBWOF2KQ53Y.0 size=1048576 2023-12-03T11:48:46.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.436+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~2EBWOF2KQ53Y.1 size=137264 2023-12-03T11:48:46.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.436+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=2EBWOF2KQ53Y tag=IJQARCCU9TN4 type=write size=2234416 2023-12-03T11:48:46.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.437+0000 7f24574f5780 1 < canceled operation key=A4LOQE7VRDN7 tag=KZS870F4WJEM type=del size=913639 2023-12-03T11:48:46.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.438+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=MPU47S1GI5YA type=write size=43180 2023-12-03T11:48:46.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.439+0000 7f24574f5780 -1 < failed to complete operation key=DA6ZXCLKTC1L tag=PXZG24LL8HUW type=del size=2486014: (22) Invalid argument 2023-12-03T11:48:46.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.441+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=XASKALSV4VN4 type=del size=1934594 2023-12-03T11:48:46.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.442+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~I56WB6ZZQHAH.0 size=1026434 2023-12-03T11:48:46.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.444+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=I56WB6ZZQHAH tag=QK4QVGF3D625 type=write size=2075010 2023-12-03T11:48:46.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.446+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.0 size=1048576 2023-12-03T11:48:46.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.447+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.1 size=1048576 2023-12-03T11:48:46.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.448+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.2 size=206844 2023-12-03T11:48:46.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.448+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=21MSE6DQQ3M6 tag=J3M7YJ1KZ5L0 type=write size=3352572 2023-12-03T11:48:46.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.449+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~85B1966F8FZI.0 size=1048576 2023-12-03T11:48:46.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.450+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~85B1966F8FZI.1 size=958048 2023-12-03T11:48:46.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.452+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=85B1966F8FZI tag=WV9F8F67Q7FR type=write size=3055200 2023-12-03T11:48:46.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.454+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.0 size=1048576 2023-12-03T11:48:46.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.455+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.1 size=1048576 2023-12-03T11:48:46.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.456+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.2 size=656669 2023-12-03T11:48:46.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.456+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=TO7J3RCV1VMB tag=VDMZUMQBMTC0 type=write size=3802397 2023-12-03T11:48:46.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.457+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~72B8DPJZ00KY.0 size=1048576 2023-12-03T11:48:46.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.458+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~72B8DPJZ00KY.1 size=107282 2023-12-03T11:48:46.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.460+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=72B8DPJZ00KY tag=PRSZID8B7IX4 type=write size=2204434 2023-12-03T11:48:46.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.461+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=AUGNQH6SQRQ1 type=del size=2649348 2023-12-03T11:48:46.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.462+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~U70KD52WH1DF.0 size=314453 2023-12-03T11:48:46.465 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.464+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=U70KD52WH1DF tag=WBF259VINXES type=write size=1363029 2023-12-03T11:48:46.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.465+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=L3WW37TTLNSH type=del size=3329780 2023-12-03T11:48:46.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.466+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.0 size=1048576 2023-12-03T11:48:46.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.467+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.1 size=1048576 2023-12-03T11:48:46.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.468+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~VI5Q7EOG7TYS.2 size=370529 2023-12-03T11:48:46.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.470+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=VI5Q7EOG7TYS tag=092DQ1LAJT93 type=write size=3516257 2023-12-03T11:48:46.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.472+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.0 size=1048576 2023-12-03T11:48:46.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.473+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.1 size=1048576 2023-12-03T11:48:46.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.473+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.2 size=859721 2023-12-03T11:48:46.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.473+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=B8YO2PV6WONG tag=BCA9PO5GRKPL type=write size=4005449 2023-12-03T11:48:46.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.475+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~LLVO6URUW41X.0 size=1048576 2023-12-03T11:48:46.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.475+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~LLVO6URUW41X.1 size=196584 2023-12-03T11:48:46.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.477+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=LLVO6URUW41X tag=L6C04S3G9SDF type=write size=2293736 2023-12-03T11:48:46.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.478+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=IAKP1QDW2SGL type=del size=830384 2023-12-03T11:48:46.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.479+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~KLGWXBCALUJE.0 size=1048576 2023-12-03T11:48:46.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.480+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~KLGWXBCALUJE.1 size=300434 2023-12-03T11:48:46.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.482+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=KLGWXBCALUJE tag=NMQT428L0XRH type=write size=2397586 2023-12-03T11:48:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.483+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=U7R4X1GDT7NI type=del size=2103939 2023-12-03T11:48:46.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.485+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=HPECYIAWBFLT type=del 2023-12-03T11:48:46.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.487+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=VP3GGEGKM4YP type=del size=2192488 2023-12-03T11:48:46.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.489+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=WLHSNKU98XG3 type=del 2023-12-03T11:48:46.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.490+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=FUEACAAXQ8D0 type=write size=476924 2023-12-03T11:48:46.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.492+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=5Y8VZYUE5PH4 type=del 2023-12-03T11:48:46.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.493+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=X7FBO79LZ081 type=del size=813066 2023-12-03T11:48:46.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.494+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=UIM1UJ087D01 type=del 2023-12-03T11:48:46.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.495+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=C0F3UDCGIUEP type=del size=3008006 2023-12-03T11:48:46.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.496+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.0 size=1048576 2023-12-03T11:48:46.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.497+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.1 size=1048576 2023-12-03T11:48:46.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.497+0000 7f24574f5780 1 < canceled part key=_multipart_ON2E6H35N9FJ.2~AW02QC3WVHXJ.2 size=583957 2023-12-03T11:48:46.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.499+0000 7f24574f5780 1 < canceled multipart upload key=ON2E6H35N9FJ upload=AW02QC3WVHXJ tag=9Y045NL5JIYW type=write size=3729685 2023-12-03T11:48:46.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.501+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~KI8WLBL8HCFQ.0 size=962479 2023-12-03T11:48:46.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.501+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=KI8WLBL8HCFQ tag=4FJQKHJ2WOU8 type=write size=2011055 2023-12-03T11:48:46.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.502+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~RBBP3BIERWWW.0 size=1048576 2023-12-03T11:48:46.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.503+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~RBBP3BIERWWW.1 size=650567 2023-12-03T11:48:46.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.505+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=RBBP3BIERWWW tag=81E7TW201S3X type=write size=2747719 2023-12-03T11:48:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.506+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=09RGNKL6REWA type=del size=605556 2023-12-03T11:48:46.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.507+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~2EBWOF2KQ53Y.0 size=1048576 2023-12-03T11:48:46.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.507+0000 7f24574f5780 1 < canceled part key=_multipart_U043YAPGUBT5.2~2EBWOF2KQ53Y.1 size=137264 2023-12-03T11:48:46.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.509+0000 7f24574f5780 1 < canceled multipart upload key=U043YAPGUBT5 upload=2EBWOF2KQ53Y tag=IJQARCCU9TN4 type=write size=2234416 2023-12-03T11:48:46.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.510+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=T1IIP1U0F1M3 type=del size=241240 2023-12-03T11:48:46.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.512+0000 7f24574f5780 1 < completed write operation key=PS1OYWI44L80 tag=MPU47S1GI5YA type=write size=43180 2023-12-03T11:48:46.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.514+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~BORLDNW5CYL2.0 size=1048576 2023-12-03T11:48:46.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.515+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~BORLDNW5CYL2.1 size=1033225 2023-12-03T11:48:46.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.515+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=BORLDNW5CYL2 tag=II1C5UTJJPO1 type=write size=3130377 2023-12-03T11:48:46.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.517+0000 7f24574f5780 1 < suggested remove operation key=S0EH0Q0UA8HG tag=XASKALSV4VN4 type=del 2023-12-03T11:48:46.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.518+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=F9DQ3U97MWL5 type=del size=2393543 2023-12-03T11:48:46.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.519+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.0 size=1048576 2023-12-03T11:48:46.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.520+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.1 size=1048576 2023-12-03T11:48:46.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.521+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~21MSE6DQQ3M6.2 size=206844 2023-12-03T11:48:46.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.524+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=21MSE6DQQ3M6 tag=J3M7YJ1KZ5L0 type=write size=3352572 2023-12-03T11:48:46.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.526+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=K9F1EK587CNG type=del size=2069053 2023-12-03T11:48:46.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.527+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.0 size=1048576 2023-12-03T11:48:46.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.528+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.1 size=1048576 2023-12-03T11:48:46.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.529+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~TO7J3RCV1VMB.2 size=656669 2023-12-03T11:48:46.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.532+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=TO7J3RCV1VMB tag=VDMZUMQBMTC0 type=write size=3802397 2023-12-03T11:48:46.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.533+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=AMW3WQJVDCIT type=write size=515022 2023-12-03T11:48:46.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.534+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=AUGNQH6SQRQ1 type=del 2023-12-03T11:48:46.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.536+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=8K2IUAGYOAPX type=del size=1381373 2023-12-03T11:48:46.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.537+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=L3WW37TTLNSH type=del 2023-12-03T11:48:46.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.538+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=5XBDA63KP67X type=del size=3784678 2023-12-03T11:48:46.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.539+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.0 size=1048576 2023-12-03T11:48:46.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.540+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.1 size=1048576 2023-12-03T11:48:46.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.541+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~B8YO2PV6WONG.2 size=859721 2023-12-03T11:48:46.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.543+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=B8YO2PV6WONG tag=BCA9PO5GRKPL type=write size=4005449 2023-12-03T11:48:46.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.544+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=0EZDS7U8F4RM type=del size=3387219 2023-12-03T11:48:46.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.546+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=IAKP1QDW2SGL type=del 2023-12-03T11:48:46.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.548+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.0 size=1048576 2023-12-03T11:48:46.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.549+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.1 size=1048576 2023-12-03T11:48:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.550+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.2 size=470428 2023-12-03T11:48:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.550+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=AHUEJGGTZB3B tag=KY8JPAWGIVU5 type=write size=3616156 2023-12-03T11:48:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.551+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=U7R4X1GDT7NI type=del 2023-12-03T11:48:46.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.552+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=JU517ZM2ZNFI type=del size=1992216 2023-12-03T11:48:46.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.553+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=VP3GGEGKM4YP type=del 2023-12-03T11:48:46.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.555+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~HWJZQ5BYIFN7.0 size=1033620 2023-12-03T11:48:46.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.555+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=HWJZQ5BYIFN7 tag=LGPAW7XFA950 type=write size=2082196 2023-12-03T11:48:46.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.557+0000 7f24574f5780 1 < suggested remove operation key=MY3K9M4QRS3L tag=FUEACAAXQ8D0 type=write 2023-12-03T11:48:46.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.559+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=FA28QR0ASF58 type=del size=1156488 2023-12-03T11:48:46.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.561+0000 7f24574f5780 1 < suggested update operation key=1V1C59DP8YPF tag=X7FBO79LZ081 type=del size=813066 2023-12-03T11:48:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.563+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~8WJYDU3WH04E.0 size=1048576 2023-12-03T11:48:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.564+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~8WJYDU3WH04E.1 size=282383 2023-12-03T11:48:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.564+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=8WJYDU3WH04E tag=TL7A505746F0 type=write size=2379535 2023-12-03T11:48:46.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.565+0000 7f24574f5780 -1 < failed to complete operation key=S0EH0Q0UA8HG tag=C0F3UDCGIUEP type=del size=3008006: (22) Invalid argument 2023-12-03T11:48:46.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.566+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=YP3V56U6LU9S type=del size=590182 2023-12-03T11:48:46.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.567+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~KI8WLBL8HCFQ.0 size=962479 2023-12-03T11:48:46.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.569+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=KI8WLBL8HCFQ tag=4FJQKHJ2WOU8 type=write size=2011055 2023-12-03T11:48:46.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.570+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=JNSGPUDQKO16 type=del size=2707108 2023-12-03T11:48:46.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.571+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=09RGNKL6REWA type=del 2023-12-03T11:48:46.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.573+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=HL10ZIU36IGR type=del size=442325 2023-12-03T11:48:46.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.574+0000 7f24574f5780 -1 < failed to complete operation key=MY3K9M4QRS3L tag=T1IIP1U0F1M3 type=del size=241240: (22) Invalid argument 2023-12-03T11:48:46.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.575+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=82Y0KGRD5Z36 type=del size=265084 2023-12-03T11:48:46.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.576+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~BORLDNW5CYL2.0 size=1048576 2023-12-03T11:48:46.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.577+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~BORLDNW5CYL2.1 size=1033225 2023-12-03T11:48:46.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.580+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=BORLDNW5CYL2 tag=II1C5UTJJPO1 type=write size=3130377 2023-12-03T11:48:46.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.581+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=XVG1TYDWCNZM type=write size=606323 2023-12-03T11:48:46.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.582+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=F9DQ3U97MWL5 type=del 2023-12-03T11:48:46.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.584+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.0 size=1048576 2023-12-03T11:48:46.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.585+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.1 size=1048576 2023-12-03T11:48:46.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.586+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.2 size=524383 2023-12-03T11:48:46.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.586+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=BSVXODZZYW0X tag=32IFZWAWC9EC type=write size=3670111 2023-12-03T11:48:46.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.587+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=K9F1EK587CNG type=del 2023-12-03T11:48:46.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.588+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=QHAA38LHNB81 type=del size=437003 2023-12-03T11:48:46.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.589+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=AMW3WQJVDCIT type=write size=515022 2023-12-03T11:48:46.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.591+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.0 size=1048576 2023-12-03T11:48:46.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.592+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.1 size=1048576 2023-12-03T11:48:46.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.593+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.2 size=642670 2023-12-03T11:48:46.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.593+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=FVSMW8SPGPXA tag=DO1GHBRR2LFG type=write size=3788398 2023-12-03T11:48:46.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.594+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=8K2IUAGYOAPX type=del 2023-12-03T11:48:46.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.595+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=V9FRLL7MIJY6 type=del size=2922894 2023-12-03T11:48:46.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.597+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=5XBDA63KP67X type=del 2023-12-03T11:48:46.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.599+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.0 size=1048576 2023-12-03T11:48:46.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.600+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.1 size=1048576 2023-12-03T11:48:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.601+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.2 size=545726 2023-12-03T11:48:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.601+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=45VNNCFAJ9QU tag=4Y8ZAFQG4213 type=write size=3691454 2023-12-03T11:48:46.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.603+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=0EZDS7U8F4RM type=del size=3387219 2023-12-03T11:48:46.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.604+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=T8AXQAC43J1T type=del size=977756 2023-12-03T11:48:46.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.605+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.0 size=1048576 2023-12-03T11:48:46.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.606+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.1 size=1048576 2023-12-03T11:48:46.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.607+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~AHUEJGGTZB3B.2 size=470428 2023-12-03T11:48:46.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.609+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=AHUEJGGTZB3B tag=KY8JPAWGIVU5 type=write size=3616156 2023-12-03T11:48:46.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.611+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~FQP05I3SSQUH.0 size=90005 2023-12-03T11:48:46.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.611+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=FQP05I3SSQUH tag=8DNV6EKMAPLE type=write size=1138581 2023-12-03T11:48:46.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.613+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=JU517ZM2ZNFI type=del 2023-12-03T11:48:46.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.615+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.0 size=1048576 2023-12-03T11:48:46.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.616+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.1 size=1048576 2023-12-03T11:48:46.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.617+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.2 size=644650 2023-12-03T11:48:46.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.617+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=GIAW64IKIK57 tag=TAA19161SIN8 type=write size=3790378 2023-12-03T11:48:46.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.618+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~HWJZQ5BYIFN7.0 size=1033620 2023-12-03T11:48:46.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.620+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=HWJZQ5BYIFN7 tag=LGPAW7XFA950 type=write size=2082196 2023-12-03T11:48:46.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.621+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=OMK3M6936API type=del size=14984 2023-12-03T11:48:46.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.622+0000 7f24574f5780 -1 < failed to complete operation key=1V1C59DP8YPF tag=FA28QR0ASF58 type=del size=1156488: (22) Invalid argument 2023-12-03T11:48:46.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.623+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=33GQOP2COYA4 type=del size=594383 2023-12-03T11:48:46.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.623+0000 7f24574f5780 1 < canceled part key=_multipart_U043YAPGUBT5.2~8WJYDU3WH04E.0 size=1048576 2023-12-03T11:48:46.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.624+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~8WJYDU3WH04E.1 size=282383 2023-12-03T11:48:46.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.627+0000 7f24574f5780 1 < canceled multipart upload key=U043YAPGUBT5 upload=8WJYDU3WH04E tag=TL7A505746F0 type=write size=2379535 2023-12-03T11:48:46.629 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.628+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=HDVDWTWQHGZG type=del size=763312 2023-12-03T11:48:46.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.629+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=YP3V56U6LU9S type=del 2023-12-03T11:48:46.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.630+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=6X7TG0FXT2P5 type=del size=1841031 2023-12-03T11:48:46.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.632+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=JNSGPUDQKO16 type=del 2023-12-03T11:48:46.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.633+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=8WC0Q6RD7SBD type=del size=2954267 2023-12-03T11:48:46.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.634+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=HL10ZIU36IGR type=del 2023-12-03T11:48:46.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.638+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~0TOTX3D91PKQ.0 size=1048576 2023-12-03T11:48:46.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.640+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~0TOTX3D91PKQ.1 size=821693 2023-12-03T11:48:46.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.640+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=0TOTX3D91PKQ tag=KIBP3A3OD3LN type=write size=2918845 2023-12-03T11:48:46.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.643+0000 7f24574f5780 1 < suggested remove operation key=WZMWCSAZX8RB tag=82Y0KGRD5Z36 type=del 2023-12-03T11:48:46.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.645+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~8F3M5ZUFOZQV.0 size=1048576 2023-12-03T11:48:46.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.646+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~8F3M5ZUFOZQV.1 size=288619 2023-12-03T11:48:46.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.646+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=8F3M5ZUFOZQV tag=4BVMMUYGVZIR type=write size=2385771 2023-12-03T11:48:46.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.647+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=XVG1TYDWCNZM type=write size=606323 2023-12-03T11:48:46.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.649+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~VK35NRFX0WRG.0 size=42673 2023-12-03T11:48:46.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.649+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=VK35NRFX0WRG tag=C1H0OWRQM2IK type=write size=1091249 2023-12-03T11:48:46.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.651+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.0 size=1048576 2023-12-03T11:48:46.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.651+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.1 size=1048576 2023-12-03T11:48:46.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.652+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~BSVXODZZYW0X.2 size=524383 2023-12-03T11:48:46.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.654+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=BSVXODZZYW0X tag=32IFZWAWC9EC type=write size=3670111 2023-12-03T11:48:46.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.655+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=B36OM4JXJD8C type=del size=3452088 2023-12-03T11:48:46.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.657+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=QHAA38LHNB81 type=del 2023-12-03T11:48:46.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.658+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=46IW8FCX048W type=del size=3593327 2023-12-03T11:48:46.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.659+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.0 size=1048576 2023-12-03T11:48:46.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.660+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.1 size=1048576 2023-12-03T11:48:46.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.662+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~FVSMW8SPGPXA.2 size=642670 2023-12-03T11:48:46.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.664+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=FVSMW8SPGPXA tag=DO1GHBRR2LFG type=write size=3788398 2023-12-03T11:48:46.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.666+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=JH3Y7RQT1OF8 type=del size=1966522 2023-12-03T11:48:46.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.667+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=V9FRLL7MIJY6 type=del 2023-12-03T11:48:46.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.668+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=3EBO7LSB29DL type=del size=3609188 2023-12-03T11:48:46.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.669+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.0 size=1048576 2023-12-03T11:48:46.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.670+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.1 size=1048576 2023-12-03T11:48:46.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.671+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~45VNNCFAJ9QU.2 size=545726 2023-12-03T11:48:46.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.673+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=45VNNCFAJ9QU tag=4Y8ZAFQG4213 type=write size=3691454 2023-12-03T11:48:46.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.675+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.0 size=1048576 2023-12-03T11:48:46.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.676+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.1 size=1048576 2023-12-03T11:48:46.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.676+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.2 size=557910 2023-12-03T11:48:46.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.676+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=D5H2DO5NQCS1 tag=VR5Z0JUSMLN9 type=write size=3703638 2023-12-03T11:48:46.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.678+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=T8AXQAC43J1T type=del 2023-12-03T11:48:46.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.680+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.0 size=1048576 2023-12-03T11:48:46.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.681+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.1 size=1048576 2023-12-03T11:48:46.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.682+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.2 size=392106 2023-12-03T11:48:46.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.682+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=JQERRKQ0VJO3 tag=ZPNS94E2QNCZ type=write size=3537834 2023-12-03T11:48:46.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.683+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~FQP05I3SSQUH.0 size=90005 2023-12-03T11:48:46.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.685+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=FQP05I3SSQUH tag=8DNV6EKMAPLE type=write size=1138581 2023-12-03T11:48:46.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.687+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~54YRGI2NMCUQ.0 size=1048576 2023-12-03T11:48:46.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.688+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~54YRGI2NMCUQ.1 size=140829 2023-12-03T11:48:46.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.688+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=54YRGI2NMCUQ tag=9IJ66MN9QCSY type=write size=2237981 2023-12-03T11:48:46.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.689+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.0 size=1048576 2023-12-03T11:48:46.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.689+0000 7f24574f5780 1 < canceled part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.1 size=1048576 2023-12-03T11:48:46.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.690+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~GIAW64IKIK57.2 size=644650 2023-12-03T11:48:46.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.692+0000 7f24574f5780 1 < canceled multipart upload key=9LKFY1U0JRBQ upload=GIAW64IKIK57 tag=TAA19161SIN8 type=write size=3790378 2023-12-03T11:48:46.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.694+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.0 size=1048576 2023-12-03T11:48:46.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.695+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.1 size=1048576 2023-12-03T11:48:46.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.696+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.2 size=251912 2023-12-03T11:48:46.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.696+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=W3E6GAH7SZOF tag=NWDNBL7PMLON type=write size=3397640 2023-12-03T11:48:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.697+0000 7f24574f5780 1 < canceled operation key=MH1SRAATMNXB tag=OMK3M6936API type=del size=14984 2023-12-03T11:48:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.698+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=V92VPPRYKF43 type=write size=987608 2023-12-03T11:48:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.700+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=33GQOP2COYA4 type=del 2023-12-03T11:48:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.701+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=4N47NJO2KWAV type=write size=729128 2023-12-03T11:48:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.702+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=HDVDWTWQHGZG type=del 2023-12-03T11:48:46.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.703+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=8BGHOEI64FOF type=write size=476189 2023-12-03T11:48:46.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.704+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=6X7TG0FXT2P5 type=del 2023-12-03T11:48:46.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.705+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=3ZD54Y7BK0HM type=write size=507468 2023-12-03T11:48:46.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.706+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=8WC0Q6RD7SBD type=del 2023-12-03T11:48:46.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.707+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=FALM0887VT28 type=del size=827181 2023-12-03T11:48:46.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.709+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~0TOTX3D91PKQ.0 size=1048576 2023-12-03T11:48:46.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.710+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~0TOTX3D91PKQ.1 size=821693 2023-12-03T11:48:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.711+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=0TOTX3D91PKQ tag=KIBP3A3OD3LN type=write size=2918845 2023-12-03T11:48:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.713+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=WV4WO73EBJ1N type=del size=3846614 2023-12-03T11:48:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.714+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~8F3M5ZUFOZQV.0 size=1048576 2023-12-03T11:48:46.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.715+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~8F3M5ZUFOZQV.1 size=288619 2023-12-03T11:48:46.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.716+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=8F3M5ZUFOZQV tag=4BVMMUYGVZIR type=write size=2385771 2023-12-03T11:48:46.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.718+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=H5V2HIC2FUUW type=del size=1501772 2023-12-03T11:48:46.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.719+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~VK35NRFX0WRG.0 size=42673 2023-12-03T11:48:46.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.720+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=VK35NRFX0WRG tag=C1H0OWRQM2IK type=write size=1091249 2023-12-03T11:48:46.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.722+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=ZZQT1A8761IZ type=del size=4019159 2023-12-03T11:48:46.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.723+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=B36OM4JXJD8C type=del 2023-12-03T11:48:46.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.724+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=PSIKHQPKFPHW type=del size=3727512 2023-12-03T11:48:46.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.725+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=46IW8FCX048W type=del 2023-12-03T11:48:46.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.727+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=59M5K81GZNPB type=write size=1005874 2023-12-03T11:48:46.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.728+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=JH3Y7RQT1OF8 type=del 2023-12-03T11:48:46.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.729+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=MVKX2EE41X1R type=del size=1930014 2023-12-03T11:48:46.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.730+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=3EBO7LSB29DL type=del 2023-12-03T11:48:46.734 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.732+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~ZVI5ITGITAV2.0 size=1048576 2023-12-03T11:48:46.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.733+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~ZVI5ITGITAV2.1 size=525385 2023-12-03T11:48:46.735 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.733+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=ZVI5ITGITAV2 tag=RH4G3VVFKNKJ type=write size=2622537 2023-12-03T11:48:46.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.735+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.0 size=1048576 2023-12-03T11:48:46.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.736+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.1 size=1048576 2023-12-03T11:48:46.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.737+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~D5H2DO5NQCS1.2 size=557910 2023-12-03T11:48:46.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.739+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=D5H2DO5NQCS1 tag=VR5Z0JUSMLN9 type=write size=3703638 2023-12-03T11:48:46.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.741+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.0 size=1048576 2023-12-03T11:48:46.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.742+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.1 size=1048576 2023-12-03T11:48:46.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.743+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.2 size=978267 2023-12-03T11:48:46.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.743+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=847IA84BG548 tag=Y2R1PIM9EVN1 type=write size=4123995 2023-12-03T11:48:46.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.744+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.0 size=1048576 2023-12-03T11:48:46.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.746+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.1 size=1048576 2023-12-03T11:48:46.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.747+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~JQERRKQ0VJO3.2 size=392106 2023-12-03T11:48:46.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.750+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=JQERRKQ0VJO3 tag=ZPNS94E2QNCZ type=write size=3537834 2023-12-03T11:48:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.751+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=0G8LR9W0LL9E type=del size=3432594 2023-12-03T11:48:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.753+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~54YRGI2NMCUQ.0 size=1048576 2023-12-03T11:48:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.754+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~54YRGI2NMCUQ.1 size=140829 2023-12-03T11:48:46.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.756+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=54YRGI2NMCUQ tag=9IJ66MN9QCSY type=write size=2237981 2023-12-03T11:48:46.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.758+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~WT4YZGLVMVZF.0 size=752173 2023-12-03T11:48:46.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.758+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=WT4YZGLVMVZF tag=E40RN4JLYK9V type=write size=1800749 2023-12-03T11:48:46.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.759+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.0 size=1048576 2023-12-03T11:48:46.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.759+0000 7f24574f5780 1 < canceled part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.1 size=1048576 2023-12-03T11:48:46.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.760+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~W3E6GAH7SZOF.2 size=251912 2023-12-03T11:48:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.762+0000 7f24574f5780 1 < canceled multipart upload key=MY3K9M4QRS3L upload=W3E6GAH7SZOF tag=NWDNBL7PMLON type=write size=3397640 2023-12-03T11:48:46.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.764+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=0RWVNGTKVOJ2 type=del size=1114445 2023-12-03T11:48:46.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.765+0000 7f24574f5780 1 < completed write operation key=1V1C59DP8YPF tag=V92VPPRYKF43 type=write size=987608 2023-12-03T11:48:46.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.766+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=QMN6S04KJYJL type=del size=2763782 2023-12-03T11:48:46.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.769+0000 7f24574f5780 1 < suggested update operation key=SC8UOVLQQ0VD tag=4N47NJO2KWAV type=write size=729128 2023-12-03T11:48:46.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.770+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=4IV0BTK1TY7P type=del size=41428 2023-12-03T11:48:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.771+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=8BGHOEI64FOF type=write size=476189 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.773+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.0 size=1048576 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.774+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.1 size=1048576 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.775+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.2 size=102295 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.775+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=DBGC0QE74J3R tag=L7KLMLKW63UM type=write size=3248023 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.776+0000 7f24574f5780 1 < completed write operation key=S0EH0Q0UA8HG tag=3ZD54Y7BK0HM type=write size=507468 2023-12-03T11:48:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.778+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~8MTJZKJ3YRK3.0 size=1048576 2023-12-03T11:48:46.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.779+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~8MTJZKJ3YRK3.1 size=1038782 2023-12-03T11:48:46.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.779+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=8MTJZKJ3YRK3 tag=I6RKXC57J053 type=write size=3135934 2023-12-03T11:48:46.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.780+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=FALM0887VT28 type=del 2023-12-03T11:48:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.781+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=IUTQ87TZN1UX type=del size=1121825 2023-12-03T11:48:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.783+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=WV4WO73EBJ1N type=del 2023-12-03T11:48:46.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.785+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~LUOTNXTSP1DP.0 size=677154 2023-12-03T11:48:46.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.785+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=LUOTNXTSP1DP tag=9K26ZDH6JO72 type=write size=1725730 2023-12-03T11:48:46.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.786+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=H5V2HIC2FUUW type=del 2023-12-03T11:48:46.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.788+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~BUZ75FCEOUAI.0 size=538707 2023-12-03T11:48:46.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.788+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=BUZ75FCEOUAI tag=G1EG9Q5F0629 type=write size=1587283 2023-12-03T11:48:46.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.789+0000 7f24574f5780 1 < completed delete operation key=PS1OYWI44L80 tag=ZZQT1A8761IZ type=del 2023-12-03T11:48:46.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.790+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=HRZB4BM4GA69 type=del size=1139262 2023-12-03T11:48:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.793+0000 7f24574f5780 1 < suggested remove operation key=1QR0TE9DXU5Z tag=PSIKHQPKFPHW type=del 2023-12-03T11:48:46.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.794+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=CAZGYZS6XGV1 type=del size=1068636 2023-12-03T11:48:46.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.795+0000 7f24574f5780 1 < completed write operation key=9LKFY1U0JRBQ tag=59M5K81GZNPB type=write size=1005874 2023-12-03T11:48:46.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.796+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=30Z2ZZQ6JZ81 type=write size=99098 2023-12-03T11:48:46.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.798+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=MVKX2EE41X1R type=del 2023-12-03T11:48:46.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.799+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=IDUAFWYJ1ULN type=del size=2365240 2023-12-03T11:48:46.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.800+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~ZVI5ITGITAV2.0 size=1048576 2023-12-03T11:48:46.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.802+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~ZVI5ITGITAV2.1 size=525385 2023-12-03T11:48:46.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.804+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=ZVI5ITGITAV2 tag=RH4G3VVFKNKJ type=write size=2622537 2023-12-03T11:48:46.806 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.805+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=TTH4D0BGL9XE type=del size=2830326 2023-12-03T11:48:46.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.806+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.0 size=1048576 2023-12-03T11:48:46.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.806+0000 7f24574f5780 1 < canceled part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.1 size=1048576 2023-12-03T11:48:46.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.807+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~847IA84BG548.2 size=978267 2023-12-03T11:48:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.809+0000 7f24574f5780 1 < canceled multipart upload key=PGTGVLK4DBN0 upload=847IA84BG548 tag=Y2R1PIM9EVN1 type=write size=4123995 2023-12-03T11:48:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.811+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=NL0RMJ3T347M type=write size=291178 2023-12-03T11:48:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.812+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=0G8LR9W0LL9E type=del 2023-12-03T11:48:46.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.814+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~LVLNIOKJ3O2A.0 size=1048576 2023-12-03T11:48:46.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.815+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~LVLNIOKJ3O2A.1 size=299714 2023-12-03T11:48:46.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.815+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=LVLNIOKJ3O2A tag=K6GVN9PSW0AZ type=write size=2396866 2023-12-03T11:48:46.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.816+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~WT4YZGLVMVZF.0 size=752173 2023-12-03T11:48:46.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.818+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=WT4YZGLVMVZF tag=E40RN4JLYK9V type=write size=1800749 2023-12-03T11:48:46.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.820+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~PFZ1Q1NQ9KNI.0 size=1017087 2023-12-03T11:48:46.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.820+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=PFZ1Q1NQ9KNI tag=TGNYI92YQ0ID type=write size=2065663 2023-12-03T11:48:46.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.823+0000 7f24574f5780 1 < suggested update operation key=PGTGVLK4DBN0 tag=0RWVNGTKVOJ2 type=del size=1114445 2023-12-03T11:48:46.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.824+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=OVSH21AWDHGV type=del size=462102 2023-12-03T11:48:46.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.825+0000 7f24574f5780 1 < canceled operation key=9LKFY1U0JRBQ tag=QMN6S04KJYJL type=del size=2763782 2023-12-03T11:48:46.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.827+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~AM7YA0CM7I10.0 size=1048576 2023-12-03T11:48:46.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.828+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~AM7YA0CM7I10.1 size=252614 2023-12-03T11:48:46.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.828+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=AM7YA0CM7I10 tag=GDG9HC24LLKZ type=write size=2349766 2023-12-03T11:48:46.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.829+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=4IV0BTK1TY7P type=del 2023-12-03T11:48:46.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.831+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~0QSDJWYT6VPR.0 size=862722 2023-12-03T11:48:46.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.831+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=0QSDJWYT6VPR tag=0HS6M6BDFPQV type=write size=1911298 2023-12-03T11:48:46.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.833+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.0 size=1048576 2023-12-03T11:48:46.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.833+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.1 size=1048576 2023-12-03T11:48:46.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.834+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~DBGC0QE74J3R.2 size=102295 2023-12-03T11:48:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.837+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=DBGC0QE74J3R tag=L7KLMLKW63UM type=write size=3248023 2023-12-03T11:48:46.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.839+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~H1ZTW7IEPXO3.0 size=562802 2023-12-03T11:48:46.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.839+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=H1ZTW7IEPXO3 tag=ZQS135620EDE type=write size=1611378 2023-12-03T11:48:46.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.840+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~8MTJZKJ3YRK3.0 size=1048576 2023-12-03T11:48:46.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.841+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~8MTJZKJ3YRK3.1 size=1038782 2023-12-03T11:48:46.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.843+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=8MTJZKJ3YRK3 tag=I6RKXC57J053 type=write size=3135934 2023-12-03T11:48:46.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.844+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=Y7HXC5O33TK5 type=del size=2554383 2023-12-03T11:48:46.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.845+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=IUTQ87TZN1UX type=del 2023-12-03T11:48:46.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.846+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=STDWAV6HWCA4 type=write size=556117 2023-12-03T11:48:46.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.847+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~LUOTNXTSP1DP.0 size=677154 2023-12-03T11:48:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.849+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=LUOTNXTSP1DP tag=9K26ZDH6JO72 type=write size=1725730 2023-12-03T11:48:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.850+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=QKNLXGDG18RM type=del size=3889946 2023-12-03T11:48:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.851+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~BUZ75FCEOUAI.0 size=538707 2023-12-03T11:48:46.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.853+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=BUZ75FCEOUAI tag=G1EG9Q5F0629 type=write size=1587283 2023-12-03T11:48:46.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.854+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=8OJP3RUKZKJK type=write size=371957 2023-12-03T11:48:46.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.855+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=HRZB4BM4GA69 type=del 2023-12-03T11:48:46.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.857+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.0 size=1048576 2023-12-03T11:48:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.858+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.1 size=1048576 2023-12-03T11:48:46.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.859+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.2 size=843456 2023-12-03T11:48:46.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.859+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=G3BJDJT4Z3H8 tag=RQ547ICBJYVV type=write size=3989184 2023-12-03T11:48:46.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.861+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=CAZGYZS6XGV1 type=del 2023-12-03T11:48:46.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.862+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=V8CVHWTOXQH1 type=del size=3802154 2023-12-03T11:48:46.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.863+0000 7f24574f5780 1 < completed write operation key=35U7YCDYMX46 tag=30Z2ZZQ6JZ81 type=write size=99098 2023-12-03T11:48:46.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.864+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=48OEG7YX3I22 type=write size=630683 2023-12-03T11:48:46.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.867+0000 7f24574f5780 1 < suggested remove operation key=35U7YCDYMX46 tag=IDUAFWYJ1ULN type=del 2023-12-03T11:48:46.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.868+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=WYTEUNKRJRZN type=del size=4143416 2023-12-03T11:48:46.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.869+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=TTH4D0BGL9XE type=del 2023-12-03T11:48:46.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.871+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=RXS69ZK2IU4N type=del size=2587180 2023-12-03T11:48:46.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.872+0000 7f24574f5780 1 < completed write operation key=XZFHFS76WNR4 tag=NL0RMJ3T347M type=write size=291178 2023-12-03T11:48:46.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.873+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=XLH2G5N1JU0R type=write size=420526 2023-12-03T11:48:46.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.874+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~LVLNIOKJ3O2A.0 size=1048576 2023-12-03T11:48:46.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.874+0000 7f24574f5780 1 < canceled part key=_multipart_PS1OYWI44L80.2~LVLNIOKJ3O2A.1 size=299714 2023-12-03T11:48:46.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.877+0000 7f24574f5780 1 < canceled multipart upload key=PS1OYWI44L80 upload=LVLNIOKJ3O2A tag=K6GVN9PSW0AZ type=write size=2396866 2023-12-03T11:48:46.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.878+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=W11D18LIR579 type=write size=87920 2023-12-03T11:48:46.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.879+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~PFZ1Q1NQ9KNI.0 size=1017087 2023-12-03T11:48:46.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.881+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=PFZ1Q1NQ9KNI tag=TGNYI92YQ0ID type=write size=2065663 2023-12-03T11:48:46.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.882+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=DOOBSHNX5D3S type=del size=2413769 2023-12-03T11:48:46.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.884+0000 7f24574f5780 1 < completed delete operation key=3EOEF9UMATIE tag=OVSH21AWDHGV type=del 2023-12-03T11:48:46.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.885+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=VKNG7I0Y0WZ2 type=write size=308938 2023-12-03T11:48:46.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.886+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~AM7YA0CM7I10.0 size=1048576 2023-12-03T11:48:46.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.887+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~AM7YA0CM7I10.1 size=252614 2023-12-03T11:48:46.890 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.889+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=AM7YA0CM7I10 tag=GDG9HC24LLKZ type=write size=2349766 2023-12-03T11:48:46.893 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.891+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.0 size=1048576 2023-12-03T11:48:46.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.892+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.1 size=1048576 2023-12-03T11:48:46.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.893+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.2 size=261691 2023-12-03T11:48:46.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.893+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=U866CO2WEHJ9 tag=5K7NTKIE4TEE type=write size=3407419 2023-12-03T11:48:46.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.895+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~0QSDJWYT6VPR.0 size=862722 2023-12-03T11:48:46.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.897+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=0QSDJWYT6VPR tag=0HS6M6BDFPQV type=write size=1911298 2023-12-03T11:48:46.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.898+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=AFL68QHKESHG type=del size=4153939 2023-12-03T11:48:46.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.899+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~H1ZTW7IEPXO3.0 size=562802 2023-12-03T11:48:46.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.901+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=H1ZTW7IEPXO3 tag=ZQS135620EDE type=write size=1611378 2023-12-03T11:48:46.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.902+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=JUXB8U6TPZCE type=write size=223516 2023-12-03T11:48:46.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.903+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=Y7HXC5O33TK5 type=del 2023-12-03T11:48:46.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.905+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~J9CQ6LE0941Z.0 size=100220 2023-12-03T11:48:46.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.905+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=J9CQ6LE0941Z tag=UTQ6KTCZ9439 type=write size=1148796 2023-12-03T11:48:46.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.906+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=STDWAV6HWCA4 type=write size=556117 2023-12-03T11:48:46.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.908+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.0 size=1048576 2023-12-03T11:48:46.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.909+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.1 size=1048576 2023-12-03T11:48:46.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.910+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.2 size=692427 2023-12-03T11:48:46.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.910+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=302DEI6L973Z tag=2KSBICCBH7QC type=write size=3838155 2023-12-03T11:48:46.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.911+0000 7f24574f5780 -1 < failed to complete operation key=35U7YCDYMX46 tag=QKNLXGDG18RM type=del size=3889946: (22) Invalid argument 2023-12-03T11:48:46.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.913+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.0 size=1048576 2023-12-03T11:48:46.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.914+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.1 size=1048576 2023-12-03T11:48:46.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.915+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.2 size=270661 2023-12-03T11:48:46.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.915+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=X4D4N6VHTIFM tag=S30MI7H9IW7T type=write size=3416389 2023-12-03T11:48:46.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.916+0000 7f24574f5780 1 < canceled operation key=HM9JHOV8W24F tag=8OJP3RUKZKJK type=write size=371957 2023-12-03T11:48:46.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.917+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=BC8OXTM0MXM6 type=del size=3781960 2023-12-03T11:48:46.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.917+0000 7f24574f5780 1 < canceled part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.0 size=1048576 2023-12-03T11:48:46.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.918+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.1 size=1048576 2023-12-03T11:48:46.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.919+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~G3BJDJT4Z3H8.2 size=843456 2023-12-03T11:48:46.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.921+0000 7f24574f5780 1 < canceled multipart upload key=35U7YCDYMX46 upload=G3BJDJT4Z3H8 tag=RQ547ICBJYVV type=write size=3989184 2023-12-03T11:48:46.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.923+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.0 size=1048576 2023-12-03T11:48:46.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.924+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.1 size=1048576 2023-12-03T11:48:46.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.925+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.2 size=864264 2023-12-03T11:48:46.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.925+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=UK6QEUZVT94R tag=6V3O8WJVY2U2 type=write size=4009992 2023-12-03T11:48:46.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.926+0000 7f24574f5780 1 < canceled operation key=XZFHFS76WNR4 tag=V8CVHWTOXQH1 type=del size=3802154 2023-12-03T11:48:46.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.927+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=DS8GI0PGHNG5 type=write size=117321 2023-12-03T11:48:46.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.930+0000 7f24574f5780 1 < suggested update operation key=A4LOQE7VRDN7 tag=48OEG7YX3I22 type=write size=630683 2023-12-03T11:48:46.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.931+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=62B90Y2UMLQG type=write size=843014 2023-12-03T11:48:46.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.932+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=WYTEUNKRJRZN type=del 2023-12-03T11:48:46.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.933+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=KO836BVRJVW6 type=del size=1971985 2023-12-03T11:48:46.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.934+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=RXS69ZK2IU4N type=del 2023-12-03T11:48:46.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.935+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=YDO8BQ3QAA8L type=del size=1182784 2023-12-03T11:48:46.940 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.939+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=XLH2G5N1JU0R type=write 2023-12-03T11:48:46.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.940+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=ABY2DP9I1XZ2 type=write size=559650 2023-12-03T11:48:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.943+0000 7f24574f5780 1 < suggested update operation key=SC8UOVLQQ0VD tag=W11D18LIR579 type=write size=87920 2023-12-03T11:48:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.944+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=HD4WC3768HT9 type=del size=510312 2023-12-03T11:48:46.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.945+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=DOOBSHNX5D3S type=del 2023-12-03T11:48:46.947 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.946+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=RA4A2WDMS3BL type=write size=362032 2023-12-03T11:48:46.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.947+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=VKNG7I0Y0WZ2 type=write size=308938 2023-12-03T11:48:46.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.949+0000 7f24574f5780 1 > prepared operation key=MH1SRAATMNXB tag=4BPPSOE5LB2W type=del size=287845 2023-12-03T11:48:46.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.950+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.0 size=1048576 2023-12-03T11:48:46.952 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.951+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.1 size=1048576 2023-12-03T11:48:46.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.952+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~U866CO2WEHJ9.2 size=261691 2023-12-03T11:48:46.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.954+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=U866CO2WEHJ9 tag=5K7NTKIE4TEE type=write size=3407419 2023-12-03T11:48:46.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.955+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=TQ8A437PZ73Z type=del size=3481462 2023-12-03T11:48:46.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.956+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=AFL68QHKESHG type=del 2023-12-03T11:48:46.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.959+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~5Q090R5HZ7FA.0 size=426033 2023-12-03T11:48:46.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.959+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=5Q090R5HZ7FA tag=VFG1J0DOFBMT type=write size=1474609 2023-12-03T11:48:46.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.960+0000 7f24574f5780 -1 < failed to complete operation key=A4LOQE7VRDN7 tag=JUXB8U6TPZCE type=write size=223516: (22) Invalid argument 2023-12-03T11:48:46.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.961+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=53ZMIIRRWUCX type=del size=742269 2023-12-03T11:48:46.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.962+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~J9CQ6LE0941Z.0 size=100220 2023-12-03T11:48:46.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.964+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=J9CQ6LE0941Z tag=UTQ6KTCZ9439 type=write size=1148796 2023-12-03T11:48:46.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.966+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~6VV0E4P2UVX6.0 size=1048576 2023-12-03T11:48:46.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.967+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~6VV0E4P2UVX6.1 size=46479 2023-12-03T11:48:46.968 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.967+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=6VV0E4P2UVX6 tag=5573VPENYNQK type=write size=2143631 2023-12-03T11:48:46.969 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.968+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.0 size=1048576 2023-12-03T11:48:46.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.969+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.1 size=1048576 2023-12-03T11:48:46.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.970+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~302DEI6L973Z.2 size=692427 2023-12-03T11:48:46.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.974+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=302DEI6L973Z tag=2KSBICCBH7QC type=write size=3838155 2023-12-03T11:48:46.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.976+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~8IFUXEJAV1JW.0 size=1048576 2023-12-03T11:48:46.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.977+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~8IFUXEJAV1JW.1 size=674540 2023-12-03T11:48:46.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.977+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=8IFUXEJAV1JW tag=7R85DDVJDNFD type=write size=2771692 2023-12-03T11:48:46.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.979+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.0 size=1048576 2023-12-03T11:48:46.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.980+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.1 size=1048576 2023-12-03T11:48:46.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.981+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~X4D4N6VHTIFM.2 size=270661 2023-12-03T11:48:46.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.983+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=X4D4N6VHTIFM tag=S30MI7H9IW7T type=write size=3416389 2023-12-03T11:48:46.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.984+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=22H27SOAGWS0 type=write size=154267 2023-12-03T11:48:46.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.987+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=BC8OXTM0MXM6 type=del 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.989+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~SL7C1AG8G1IO.0 size=1048576 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.990+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~SL7C1AG8G1IO.1 size=71227 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.990+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=SL7C1AG8G1IO tag=EONH9X85OWK6 type=write size=2168379 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.991+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.0 size=1048576 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.992+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.1 size=1048576 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.993+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~UK6QEUZVT94R.2 size=864264 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.995+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=UK6QEUZVT94R tag=6V3O8WJVY2U2 type=write size=4009992 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.997+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.0 size=1048576 2023-12-03T11:48:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.998+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.1 size=1048576 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.999+0000 7f24574f5780 1 > prepared part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.2 size=7448 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:46.999+0000 7f24574f5780 1 > prepared multipart upload key=35U7YCDYMX46 upload=50QETBERS1H7 tag=JOX8VSKD4HGS type=write size=3153176 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.000+0000 7f24574f5780 1 < completed write operation key=BMKG19L35HLV tag=DS8GI0PGHNG5 type=write size=117321 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.001+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=ZCJ5KV1WB4ZZ type=del size=2266973 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.004+0000 7f24574f5780 1 < suggested remove operation key=BV8C0VSJ1GTW tag=62B90Y2UMLQG type=write 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.005+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=PSGZFFHUCA9A type=del size=3408968 2023-12-03T11:48:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.006+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=KO836BVRJVW6 type=del 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.008+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~XEEYQBW1JMBH.0 size=516922 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.008+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=XEEYQBW1JMBH tag=PIRXBX8C96TE type=write size=1565498 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.009+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=YDO8BQ3QAA8L type=del 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.011+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~C62AYRYUY0L8.0 size=254375 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.011+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=C62AYRYUY0L8 tag=SMNHDYHIH10O type=write size=1302951 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.053+0000 7f24574f5780 1 < completed write operation key=MY3K9M4QRS3L tag=ABY2DP9I1XZ2 type=write size=559650 2023-12-03T11:48:47.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.056+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~JIGJ3WIHXAQR.0 size=1048576 2023-12-03T11:48:47.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.057+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~JIGJ3WIHXAQR.1 size=541699 2023-12-03T11:48:47.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.057+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=JIGJ3WIHXAQR tag=BGWZK2E9UUMU type=write size=2638851 2023-12-03T11:48:47.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.058+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=HD4WC3768HT9 type=del 2023-12-03T11:48:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.060+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~7HDLU9HU30Y7.0 size=355201 2023-12-03T11:48:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.060+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=7HDLU9HU30Y7 tag=UZ3URLV02AE3 type=write size=1403777 2023-12-03T11:48:47.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.061+0000 7f24574f5780 1 < canceled operation key=XRHVYVADRWET tag=RA4A2WDMS3BL type=write size=362032 2023-12-03T11:48:47.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.063+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=04AN20BK37U0 type=write size=775227 2023-12-03T11:48:47.066 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.064+0000 7f24574f5780 1 < completed delete operation key=MH1SRAATMNXB tag=4BPPSOE5LB2W type=del 2023-12-03T11:48:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.066+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=RYOGNT0BJNKN type=del size=2241627 2023-12-03T11:48:47.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.067+0000 7f24574f5780 1 < completed delete operation key=XRHVYVADRWET tag=TQ8A437PZ73Z type=del 2023-12-03T11:48:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.068+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=5QC0JU5KOCCP type=del size=799839 2023-12-03T11:48:47.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.069+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~5Q090R5HZ7FA.0 size=426033 2023-12-03T11:48:47.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.071+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=5Q090R5HZ7FA tag=VFG1J0DOFBMT type=write size=1474609 2023-12-03T11:48:47.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.073+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~Q3LEWBR02P7P.0 size=314583 2023-12-03T11:48:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.073+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=Q3LEWBR02P7P tag=L5QVBJYR4C7Z type=write size=1363159 2023-12-03T11:48:47.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.074+0000 7f24574f5780 1 < completed delete operation key=1V39QA10M1HY tag=53ZMIIRRWUCX type=del 2023-12-03T11:48:47.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.076+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.0 size=1048576 2023-12-03T11:48:47.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.077+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.1 size=1048576 2023-12-03T11:48:47.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.078+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.2 size=210536 2023-12-03T11:48:47.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.078+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=LDZ1WHPAX47U tag=EZTFMA4Q09TB type=write size=3356264 2023-12-03T11:48:47.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.079+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~6VV0E4P2UVX6.0 size=1048576 2023-12-03T11:48:47.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.080+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~6VV0E4P2UVX6.1 size=46479 2023-12-03T11:48:47.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.082+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=6VV0E4P2UVX6 tag=5573VPENYNQK type=write size=2143631 2023-12-03T11:48:47.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.083+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=MJMRTNIDL0CA type=del size=436906 2023-12-03T11:48:47.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.084+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~8IFUXEJAV1JW.0 size=1048576 2023-12-03T11:48:47.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.085+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~8IFUXEJAV1JW.1 size=674540 2023-12-03T11:48:47.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.087+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=8IFUXEJAV1JW tag=7R85DDVJDNFD type=write size=2771692 2023-12-03T11:48:47.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.088+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=TWW3N6PJGEBO type=del size=579618 2023-12-03T11:48:47.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.089+0000 7f24574f5780 1 < canceled operation key=1QR0TE9DXU5Z tag=22H27SOAGWS0 type=write size=154267 2023-12-03T11:48:47.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.091+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~XWWIRHDB83KW.0 size=1048576 2023-12-03T11:48:47.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.092+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~XWWIRHDB83KW.1 size=649655 2023-12-03T11:48:47.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.092+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=XWWIRHDB83KW tag=SCU4Y01D42OO type=write size=2746807 2023-12-03T11:48:47.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.094+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~SL7C1AG8G1IO.0 size=1048576 2023-12-03T11:48:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.095+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~SL7C1AG8G1IO.1 size=71227 2023-12-03T11:48:47.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.097+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=SL7C1AG8G1IO tag=EONH9X85OWK6 type=write size=2168379 2023-12-03T11:48:47.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.099+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.0 size=1048576 2023-12-03T11:48:47.101 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.100+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.1 size=1048576 2023-12-03T11:48:47.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.101+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.2 size=365479 2023-12-03T11:48:47.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.101+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=D18ETLUHA182 tag=TSF0P259IPE4 type=write size=3511207 2023-12-03T11:48:47.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.102+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.0 size=1048576 2023-12-03T11:48:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.103+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.1 size=1048576 2023-12-03T11:48:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.104+0000 7f24574f5780 1 < completed part key=_multipart_35U7YCDYMX46.2~50QETBERS1H7.2 size=7448 2023-12-03T11:48:47.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.106+0000 7f24574f5780 1 < completed multipart upload key=35U7YCDYMX46 upload=50QETBERS1H7 tag=JOX8VSKD4HGS type=write size=3153176 2023-12-03T11:48:47.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.109+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PDKS3RPEEM11.0 size=1048576 2023-12-03T11:48:47.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.109+0000 7f24574f5780 1 > prepared part key=_multipart_SC8UOVLQQ0VD.2~PDKS3RPEEM11.1 size=958809 2023-12-03T11:48:47.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.109+0000 7f24574f5780 1 > prepared multipart upload key=SC8UOVLQQ0VD upload=PDKS3RPEEM11 tag=RXS4MFS90NBD type=write size=3055961 2023-12-03T11:48:47.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.111+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=ZCJ5KV1WB4ZZ type=del 2023-12-03T11:48:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.113+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~0F4NOFSPEVPR.0 size=896978 2023-12-03T11:48:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.113+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=0F4NOFSPEVPR tag=IW59HEAO7TKX type=write size=1945554 2023-12-03T11:48:47.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.114+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=PSGZFFHUCA9A type=del 2023-12-03T11:48:47.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.115+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=DKSA88Z2WGI4 type=del size=2458259 2023-12-03T11:48:47.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.117+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~XEEYQBW1JMBH.0 size=516922 2023-12-03T11:48:47.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.119+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=XEEYQBW1JMBH tag=PIRXBX8C96TE type=write size=1565498 2023-12-03T11:48:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.120+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=67UAL510O9D0 type=del size=4079159 2023-12-03T11:48:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.121+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~C62AYRYUY0L8.0 size=254375 2023-12-03T11:48:47.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.123+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=C62AYRYUY0L8 tag=SMNHDYHIH10O type=write size=1302951 2023-12-03T11:48:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.126+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.0 size=1048576 2023-12-03T11:48:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.127+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.1 size=1048576 2023-12-03T11:48:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.128+0000 7f24574f5780 1 > prepared part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.2 size=8980 2023-12-03T11:48:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.128+0000 7f24574f5780 1 > prepared multipart upload key=HM9JHOV8W24F upload=W01WJ2MSTYOA tag=SFSXX7T12WX6 type=write size=3154708 2023-12-03T11:48:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.129+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~JIGJ3WIHXAQR.0 size=1048576 2023-12-03T11:48:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.129+0000 7f24574f5780 1 < canceled part key=_multipart_WZMWCSAZX8RB.2~JIGJ3WIHXAQR.1 size=541699 2023-12-03T11:48:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.131+0000 7f24574f5780 1 < canceled multipart upload key=WZMWCSAZX8RB upload=JIGJ3WIHXAQR tag=BGWZK2E9UUMU type=write size=2638851 2023-12-03T11:48:47.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.133+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~XF7JUGI5PRQG.0 size=338898 2023-12-03T11:48:47.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.133+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=XF7JUGI5PRQG tag=F89FGICP198F type=write size=1387474 2023-12-03T11:48:47.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.134+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~7HDLU9HU30Y7.0 size=355201 2023-12-03T11:48:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.136+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=7HDLU9HU30Y7 tag=UZ3URLV02AE3 type=write size=1403777 2023-12-03T11:48:47.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.137+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=JP4YD3NRMNKB type=write size=361727 2023-12-03T11:48:47.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.139+0000 7f24574f5780 1 < completed write operation key=SUBILSL92D96 tag=04AN20BK37U0 type=write size=775227 2023-12-03T11:48:47.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.140+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=IBMV76E16CWP type=del size=2375318 2023-12-03T11:48:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.141+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=RYOGNT0BJNKN type=del 2023-12-03T11:48:47.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.142+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=FQCQP446BSDO type=del size=1964053 2023-12-03T11:48:47.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.144+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=5QC0JU5KOCCP type=del 2023-12-03T11:48:47.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.145+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=7KD3K2LUYMBM type=del size=3863823 2023-12-03T11:48:47.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.146+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~Q3LEWBR02P7P.0 size=314583 2023-12-03T11:48:47.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.148+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=Q3LEWBR02P7P tag=L5QVBJYR4C7Z type=write size=1363159 2023-12-03T11:48:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.150+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~BDHNI0V38P0J.0 size=1048576 2023-12-03T11:48:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.151+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~BDHNI0V38P0J.1 size=138949 2023-12-03T11:48:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.151+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=BDHNI0V38P0J tag=NUR2KC0R72FJ type=write size=2236101 2023-12-03T11:48:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.152+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.0 size=1048576 2023-12-03T11:48:47.154 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.153+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.1 size=1048576 2023-12-03T11:48:47.155 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.154+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~LDZ1WHPAX47U.2 size=210536 2023-12-03T11:48:47.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.156+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=LDZ1WHPAX47U tag=EZTFMA4Q09TB type=write size=3356264 2023-12-03T11:48:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.158+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~6LB11TES36LU.0 size=382381 2023-12-03T11:48:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.158+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=6LB11TES36LU tag=9OLY4QI3FPTM type=write size=1430957 2023-12-03T11:48:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.160+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=MJMRTNIDL0CA type=del 2023-12-03T11:48:47.162 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.161+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=RGHS341OZRI4 type=del size=386808 2023-12-03T11:48:47.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.162+0000 7f24574f5780 1 < completed delete operation key=SUBILSL92D96 tag=TWW3N6PJGEBO type=del 2023-12-03T11:48:47.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.163+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=J0UOO71MPAJ7 type=del size=2666653 2023-12-03T11:48:47.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.165+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~XWWIRHDB83KW.0 size=1048576 2023-12-03T11:48:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.166+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~XWWIRHDB83KW.1 size=649655 2023-12-03T11:48:47.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.168+0000 7f24574f5780 1 < completed multipart upload key=PGTGVLK4DBN0 upload=XWWIRHDB83KW tag=SCU4Y01D42OO type=write size=2746807 2023-12-03T11:48:47.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.169+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=7U87BMT4X0XD type=del size=2734348 2023-12-03T11:48:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.170+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.0 size=1048576 2023-12-03T11:48:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.171+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.1 size=1048576 2023-12-03T11:48:47.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.172+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~D18ETLUHA182.2 size=365479 2023-12-03T11:48:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.174+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=D18ETLUHA182 tag=TSF0P259IPE4 type=write size=3511207 2023-12-03T11:48:47.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.176+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=RFEFY1R4HZ1H type=del size=1067049 2023-12-03T11:48:47.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.177+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PDKS3RPEEM11.0 size=1048576 2023-12-03T11:48:47.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.178+0000 7f24574f5780 1 < completed part key=_multipart_SC8UOVLQQ0VD.2~PDKS3RPEEM11.1 size=958809 2023-12-03T11:48:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.180+0000 7f24574f5780 1 < completed multipart upload key=SC8UOVLQQ0VD upload=PDKS3RPEEM11 tag=RXS4MFS90NBD type=write size=3055961 2023-12-03T11:48:47.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.181+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=V5YDHMV1MCM7 type=del size=3082571 2023-12-03T11:48:47.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.182+0000 7f24574f5780 1 < canceled part key=_multipart_LS4OEAGZWQXC.2~0F4NOFSPEVPR.0 size=896978 2023-12-03T11:48:47.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.183+0000 7f24574f5780 1 < canceled multipart upload key=LS4OEAGZWQXC upload=0F4NOFSPEVPR tag=IW59HEAO7TKX type=write size=1945554 2023-12-03T11:48:47.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.185+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~KJNLVSKVGN3J.0 size=1048576 2023-12-03T11:48:47.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.186+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~KJNLVSKVGN3J.1 size=559874 2023-12-03T11:48:47.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.186+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=KJNLVSKVGN3J tag=A92EEMSKX1Y5 type=write size=2657026 2023-12-03T11:48:47.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.188+0000 7f24574f5780 1 < suggested update operation key=BV8C0VSJ1GTW tag=DKSA88Z2WGI4 type=del size=2458259 2023-12-03T11:48:47.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.190+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.0 size=1048576 2023-12-03T11:48:47.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.191+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.1 size=1048576 2023-12-03T11:48:47.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.192+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.2 size=475048 2023-12-03T11:48:47.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.192+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=R7TOI7Z6N095 tag=FKNJ5UWTQN61 type=write size=3620776 2023-12-03T11:48:47.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.193+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=67UAL510O9D0 type=del 2023-12-03T11:48:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.194+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=NVUCN5GEWWO8 type=write size=723753 2023-12-03T11:48:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.196+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.0 size=1048576 2023-12-03T11:48:47.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.197+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.1 size=1048576 2023-12-03T11:48:47.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.198+0000 7f24574f5780 1 < completed part key=_multipart_HM9JHOV8W24F.2~W01WJ2MSTYOA.2 size=8980 2023-12-03T11:48:47.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.200+0000 7f24574f5780 1 < completed multipart upload key=HM9JHOV8W24F upload=W01WJ2MSTYOA tag=SFSXX7T12WX6 type=write size=3154708 2023-12-03T11:48:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.202+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=I88R4KQPO47Y type=del size=534421 2023-12-03T11:48:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.203+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~XF7JUGI5PRQG.0 size=338898 2023-12-03T11:48:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.205+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=XF7JUGI5PRQG tag=F89FGICP198F type=write size=1387474 2023-12-03T11:48:47.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.206+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=RSFGP9F21TB2 type=write size=772831 2023-12-03T11:48:47.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.207+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=JP4YD3NRMNKB type=write size=361727 2023-12-03T11:48:47.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.208+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=7EBB35A23MS7 type=del size=3268127 2023-12-03T11:48:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.211+0000 7f24574f5780 1 < suggested update operation key=35U7YCDYMX46 tag=IBMV76E16CWP type=del size=2375318 2023-12-03T11:48:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.213+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.0 size=1048576 2023-12-03T11:48:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.214+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.1 size=1048576 2023-12-03T11:48:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.215+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.2 size=693691 2023-12-03T11:48:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.215+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=34DTRQVZZVCR tag=A908B7W60ZS6 type=write size=3839419 2023-12-03T11:48:47.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.216+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=FQCQP446BSDO type=del 2023-12-03T11:48:47.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.217+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=2G5YYWMM1Z98 type=del size=3452109 2023-12-03T11:48:47.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.218+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=7KD3K2LUYMBM type=del 2023-12-03T11:48:47.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.219+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=TTBBVX79E3CB type=del size=2833721 2023-12-03T11:48:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.220+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~BDHNI0V38P0J.0 size=1048576 2023-12-03T11:48:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.221+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~BDHNI0V38P0J.1 size=138949 2023-12-03T11:48:47.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.223+0000 7f24574f5780 1 < completed multipart upload key=SUBILSL92D96 upload=BDHNI0V38P0J tag=NUR2KC0R72FJ type=write size=2236101 2023-12-03T11:48:47.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.224+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=ATM0FOQLUS2E type=del size=791550 2023-12-03T11:48:47.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.226+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~6LB11TES36LU.0 size=382381 2023-12-03T11:48:47.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.227+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=6LB11TES36LU tag=9OLY4QI3FPTM type=write size=1430957 2023-12-03T11:48:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.229+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~3ULKWSCTZMVF.0 size=747159 2023-12-03T11:48:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.229+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=3ULKWSCTZMVF tag=B4SH7FDEL9OD type=write size=1795735 2023-12-03T11:48:47.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.231+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=RGHS341OZRI4 type=del 2023-12-03T11:48:47.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.232+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=17J8JT07MDQ9 type=del size=3269634 2023-12-03T11:48:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.234+0000 7f24574f5780 1 < suggested update operation key=S73M6UFMX5NH tag=J0UOO71MPAJ7 type=del size=2666653 2023-12-03T11:48:47.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.237+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.0 size=1048576 2023-12-03T11:48:47.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.240+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.1 size=1048576 2023-12-03T11:48:47.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.241+0000 7f24574f5780 1 > prepared part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.2 size=942884 2023-12-03T11:48:47.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.241+0000 7f24574f5780 1 > prepared multipart upload key=9LKFY1U0JRBQ upload=BFJSLCJVEOKZ tag=RGGOJVEMAZK7 type=write size=4088612 2023-12-03T11:48:47.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.242+0000 7f24574f5780 1 < completed delete operation key=9LKFY1U0JRBQ tag=7U87BMT4X0XD type=del 2023-12-03T11:48:47.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.244+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=I9CA2X6F573W type=del size=3169863 2023-12-03T11:48:47.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.246+0000 7f24574f5780 1 < suggested update operation key=PS1OYWI44L80 tag=RFEFY1R4HZ1H type=del size=1067049 2023-12-03T11:48:47.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.247+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=RXSM7KPHEUIC type=del size=2670470 2023-12-03T11:48:47.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.248+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=V5YDHMV1MCM7 type=del 2023-12-03T11:48:47.251 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.249+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=6LALFQHO7XBI type=del size=3177166 2023-12-03T11:48:47.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.251+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~KJNLVSKVGN3J.0 size=1048576 2023-12-03T11:48:47.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.252+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~KJNLVSKVGN3J.1 size=559874 2023-12-03T11:48:47.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.254+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=KJNLVSKVGN3J tag=A92EEMSKX1Y5 type=write size=2657026 2023-12-03T11:48:47.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.256+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.0 size=1048576 2023-12-03T11:48:47.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.257+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.1 size=1048576 2023-12-03T11:48:47.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.257+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.2 size=1030937 2023-12-03T11:48:47.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.257+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=III4VGAIZUMT tag=KXZB90URFSZC type=write size=4176665 2023-12-03T11:48:47.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.259+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.0 size=1048576 2023-12-03T11:48:47.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.260+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.1 size=1048576 2023-12-03T11:48:47.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.260+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~R7TOI7Z6N095.2 size=475048 2023-12-03T11:48:47.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.263+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=R7TOI7Z6N095 tag=FKNJ5UWTQN61 type=write size=3620776 2023-12-03T11:48:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.264+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=VX4RPJHB9UGH type=del size=2901357 2023-12-03T11:48:47.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.265+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=NVUCN5GEWWO8 type=write size=723753 2023-12-03T11:48:47.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.267+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~AG8SALLY13OV.0 size=983881 2023-12-03T11:48:47.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.267+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=AG8SALLY13OV tag=5OBU1I24ZLX1 type=write size=2032457 2023-12-03T11:48:47.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.268+0000 7f24574f5780 1 < completed delete operation key=A4LOQE7VRDN7 tag=I88R4KQPO47Y type=del 2023-12-03T11:48:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.269+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=LNFPM9OEFDBP type=write size=572937 2023-12-03T11:48:47.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.270+0000 7f24574f5780 1 < completed write operation key=HM9JHOV8W24F tag=RSFGP9F21TB2 type=write size=772831 2023-12-03T11:48:47.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.272+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=2M3Z2KUJON9J type=del size=621585 2023-12-03T11:48:47.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.273+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=7EBB35A23MS7 type=del 2023-12-03T11:48:47.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.274+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=BGMW6I9PG1Z0 type=del size=806155 2023-12-03T11:48:47.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.274+0000 7f24574f5780 1 < canceled part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.0 size=1048576 2023-12-03T11:48:47.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.275+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.1 size=1048576 2023-12-03T11:48:47.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.276+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~34DTRQVZZVCR.2 size=693691 2023-12-03T11:48:47.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.278+0000 7f24574f5780 1 < canceled multipart upload key=XO42YZR60KQE upload=34DTRQVZZVCR tag=A908B7W60ZS6 type=write size=3839419 2023-12-03T11:48:47.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.281+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~4ZQHXWP6CH11.0 size=1048576 2023-12-03T11:48:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.282+0000 7f24574f5780 1 > prepared part key=_multipart_QVLPCWC4T8N8.2~4ZQHXWP6CH11.1 size=763413 2023-12-03T11:48:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.282+0000 7f24574f5780 1 > prepared multipart upload key=QVLPCWC4T8N8 upload=4ZQHXWP6CH11 tag=0S20S686MM8P type=write size=2860565 2023-12-03T11:48:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.284+0000 7f24574f5780 1 < suggested update operation key=WZMWCSAZX8RB tag=2G5YYWMM1Z98 type=del size=3452109 2023-12-03T11:48:47.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.285+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=T2HJSCOE2IHZ type=del size=2633892 2023-12-03T11:48:47.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.287+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=TTBBVX79E3CB type=del 2023-12-03T11:48:47.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.288+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=RHU40RE39VYR type=del size=3397734 2023-12-03T11:48:47.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.289+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=ATM0FOQLUS2E type=del 2023-12-03T11:48:47.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.290+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=YA0DCBK8EHW8 type=del size=2012519 2023-12-03T11:48:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.291+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~3ULKWSCTZMVF.0 size=747159 2023-12-03T11:48:47.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.293+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=3ULKWSCTZMVF tag=B4SH7FDEL9OD type=write size=1795735 2023-12-03T11:48:47.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.294+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=3XQPYTC8A2BO type=del size=3591746 2023-12-03T11:48:47.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.296+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=17J8JT07MDQ9 type=del 2023-12-03T11:48:47.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.298+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~FQSPBWXW43SD.0 size=408675 2023-12-03T11:48:47.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.298+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=FQSPBWXW43SD tag=U7M7HSYEXEIA type=write size=1457251 2023-12-03T11:48:47.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.299+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.0 size=1048576 2023-12-03T11:48:47.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.300+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.1 size=1048576 2023-12-03T11:48:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.301+0000 7f24574f5780 1 < completed part key=_multipart_9LKFY1U0JRBQ.2~BFJSLCJVEOKZ.2 size=942884 2023-12-03T11:48:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.303+0000 7f24574f5780 1 < completed multipart upload key=9LKFY1U0JRBQ upload=BFJSLCJVEOKZ tag=RGGOJVEMAZK7 type=write size=4088612 2023-12-03T11:48:47.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.304+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=0CFS07P14HKT type=del size=1230911 2023-12-03T11:48:47.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.305+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=I9CA2X6F573W type=del 2023-12-03T11:48:47.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.307+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.0 size=1048576 2023-12-03T11:48:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.308+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.1 size=1048576 2023-12-03T11:48:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.309+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.2 size=364071 2023-12-03T11:48:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.309+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=PEZ2WYI0UWL2 tag=AJO82SD7LZA2 type=write size=3509799 2023-12-03T11:48:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.310+0000 7f24574f5780 -1 < failed to complete operation key=WZMWCSAZX8RB tag=RXSM7KPHEUIC type=del size=2670470: (22) Invalid argument 2023-12-03T11:48:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.312+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=NE80YQ3Z1ZUU type=del size=1664724 2023-12-03T11:48:47.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.313+0000 7f24574f5780 1 < completed delete operation key=QVLPCWC4T8N8 tag=6LALFQHO7XBI type=del 2023-12-03T11:48:47.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.315+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~3G322WDFEY2V.0 size=592113 2023-12-03T11:48:47.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.315+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=3G322WDFEY2V tag=4OU9J7OVOX8Y type=write size=1640689 2023-12-03T11:48:47.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.316+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.0 size=1048576 2023-12-03T11:48:47.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.317+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.1 size=1048576 2023-12-03T11:48:47.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.318+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~III4VGAIZUMT.2 size=1030937 2023-12-03T11:48:47.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.320+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=III4VGAIZUMT tag=KXZB90URFSZC type=write size=4176665 2023-12-03T11:48:47.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.321+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=0IC80U19IIH2 type=write size=834672 2023-12-03T11:48:47.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.323+0000 7f24574f5780 1 < canceled operation key=S7Q5TCM499H1 tag=VX4RPJHB9UGH type=del size=2901357 2023-12-03T11:48:47.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.325+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~YGQ8GUEDLGAK.0 size=1048576 2023-12-03T11:48:47.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.326+0000 7f24574f5780 1 > prepared part key=_multipart_MY3K9M4QRS3L.2~YGQ8GUEDLGAK.1 size=533835 2023-12-03T11:48:47.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.326+0000 7f24574f5780 1 > prepared multipart upload key=MY3K9M4QRS3L upload=YGQ8GUEDLGAK tag=943ZC18ZRQBF type=write size=2630987 2023-12-03T11:48:47.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.327+0000 7f24574f5780 1 < canceled part key=_multipart_1QR0TE9DXU5Z.2~AG8SALLY13OV.0 size=983881 2023-12-03T11:48:47.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.329+0000 7f24574f5780 1 < canceled multipart upload key=1QR0TE9DXU5Z upload=AG8SALLY13OV tag=5OBU1I24ZLX1 type=write size=2032457 2023-12-03T11:48:47.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.331+0000 7f24574f5780 1 > prepared part key=_multipart_U043YAPGUBT5.2~XHY2S3D29MP8.0 size=734781 2023-12-03T11:48:47.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.331+0000 7f24574f5780 1 > prepared multipart upload key=U043YAPGUBT5 upload=XHY2S3D29MP8 tag=Y08RTNT6UB1P type=write size=1783357 2023-12-03T11:48:47.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.334+0000 7f24574f5780 1 < suggested update operation key=XRHVYVADRWET tag=LNFPM9OEFDBP type=write size=572937 2023-12-03T11:48:47.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.335+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=O9KH4N1XGTVM type=del size=1237117 2023-12-03T11:48:47.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.336+0000 7f24574f5780 1 < completed delete operation key=6WY91DFIRUN8 tag=2M3Z2KUJON9J type=del 2023-12-03T11:48:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.339+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~XM2N2LT3Z267.0 size=778861 2023-12-03T11:48:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.339+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=XM2N2LT3Z267 tag=SJYM8552K5YM type=write size=1827437 2023-12-03T11:48:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.340+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=BGMW6I9PG1Z0 type=del 2023-12-03T11:48:47.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.342+0000 7f24574f5780 1 > prepared operation key=1V39QA10M1HY tag=7U01UIV8FLFZ type=write size=682248 2023-12-03T11:48:47.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.343+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~4ZQHXWP6CH11.0 size=1048576 2023-12-03T11:48:47.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.344+0000 7f24574f5780 1 < completed part key=_multipart_QVLPCWC4T8N8.2~4ZQHXWP6CH11.1 size=763413 2023-12-03T11:48:47.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.346+0000 7f24574f5780 1 < completed multipart upload key=QVLPCWC4T8N8 upload=4ZQHXWP6CH11 tag=0S20S686MM8P type=write size=2860565 2023-12-03T11:48:47.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.347+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=R2OFMM9FKUL1 type=del size=2041960 2023-12-03T11:48:47.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.348+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=T2HJSCOE2IHZ type=del 2023-12-03T11:48:47.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.349+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=GKHGFECQ5VO8 type=del size=3084543 2023-12-03T11:48:47.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.352+0000 7f24574f5780 1 < completed delete operation key=WD7U31FW3NGZ tag=RHU40RE39VYR type=del 2023-12-03T11:48:47.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.354+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=05VDCU488P71 type=del size=1717439 2023-12-03T11:48:47.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.357+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=YA0DCBK8EHW8 type=del 2023-12-03T11:48:47.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.358+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=4IOPWNXDZ3E2 type=del size=2477531 2023-12-03T11:48:47.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.359+0000 7f24574f5780 1 < completed delete operation key=DA6ZXCLKTC1L tag=3XQPYTC8A2BO type=del 2023-12-03T11:48:47.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.363+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~HNPYFDP7UWCZ.0 size=895902 2023-12-03T11:48:47.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.363+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=HNPYFDP7UWCZ tag=009NNA1EYKAG type=write size=1944478 2023-12-03T11:48:47.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.365+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~FQSPBWXW43SD.0 size=408675 2023-12-03T11:48:47.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.367+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=FQSPBWXW43SD tag=U7M7HSYEXEIA type=write size=1457251 2023-12-03T11:48:47.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.368+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=Q1Y4RTS9JX5F type=del size=2011513 2023-12-03T11:48:47.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.369+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=0CFS07P14HKT type=del 2023-12-03T11:48:47.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.371+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~PVQB0Z90J604.0 size=413848 2023-12-03T11:48:47.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.371+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=PVQB0Z90J604 tag=09OQJ3L7VEAP type=write size=1462424 2023-12-03T11:48:47.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.372+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.0 size=1048576 2023-12-03T11:48:47.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.373+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.1 size=1048576 2023-12-03T11:48:47.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.374+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~PEZ2WYI0UWL2.2 size=364071 2023-12-03T11:48:47.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.376+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=PEZ2WYI0UWL2 tag=AJO82SD7LZA2 type=write size=3509799 2023-12-03T11:48:47.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.377+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=EDETX3ETB2QO type=write size=923774 2023-12-03T11:48:47.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.378+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=NE80YQ3Z1ZUU type=del 2023-12-03T11:48:47.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.379+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=7IN9BH8Y1ATR type=del size=322935 2023-12-03T11:48:47.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.381+0000 7f24574f5780 1 < completed part key=_multipart_XZFHFS76WNR4.2~3G322WDFEY2V.0 size=592113 2023-12-03T11:48:47.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.383+0000 7f24574f5780 1 < completed multipart upload key=XZFHFS76WNR4 upload=3G322WDFEY2V tag=4OU9J7OVOX8Y type=write size=1640689 2023-12-03T11:48:47.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.385+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.0 size=1048576 2023-12-03T11:48:47.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.386+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.1 size=1048576 2023-12-03T11:48:47.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.387+0000 7f24574f5780 1 > prepared part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.2 size=302116 2023-12-03T11:48:47.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.387+0000 7f24574f5780 1 > prepared multipart upload key=MH1SRAATMNXB upload=TF4QRBGE3KDM tag=BE1Y9SF9O4QT type=write size=3447844 2023-12-03T11:48:47.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.388+0000 7f24574f5780 1 < completed write operation key=S0EH0Q0UA8HG tag=0IC80U19IIH2 type=write size=834672 2023-12-03T11:48:47.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.389+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=1622EYNCAB8H type=del size=3752164 2023-12-03T11:48:47.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.390+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~YGQ8GUEDLGAK.0 size=1048576 2023-12-03T11:48:47.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.391+0000 7f24574f5780 1 < completed part key=_multipart_MY3K9M4QRS3L.2~YGQ8GUEDLGAK.1 size=533835 2023-12-03T11:48:47.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.393+0000 7f24574f5780 1 < completed multipart upload key=MY3K9M4QRS3L upload=YGQ8GUEDLGAK tag=943ZC18ZRQBF type=write size=2630987 2023-12-03T11:48:47.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.395+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.0 size=1048576 2023-12-03T11:48:47.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.396+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.1 size=1048576 2023-12-03T11:48:47.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.397+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.2 size=667189 2023-12-03T11:48:47.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.397+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=PS07QBT6WE8Z tag=AUBFIIBCMV9I type=write size=3812917 2023-12-03T11:48:47.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.398+0000 7f24574f5780 1 < completed part key=_multipart_U043YAPGUBT5.2~XHY2S3D29MP8.0 size=734781 2023-12-03T11:48:47.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.400+0000 7f24574f5780 1 < completed multipart upload key=U043YAPGUBT5 upload=XHY2S3D29MP8 tag=Y08RTNT6UB1P type=write size=1783357 2023-12-03T11:48:47.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.402+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~JZIP1NYQWPXF.0 size=1048576 2023-12-03T11:48:47.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.403+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~JZIP1NYQWPXF.1 size=156717 2023-12-03T11:48:47.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.403+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=JZIP1NYQWPXF tag=JLD25AUE1VQR type=write size=2253869 2023-12-03T11:48:47.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.405+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=O9KH4N1XGTVM type=del 2023-12-03T11:48:47.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.406+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=VPZD4V7EW4DQ type=del size=2054455 2023-12-03T11:48:47.410 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.408+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~XM2N2LT3Z267.0 size=778861 2023-12-03T11:48:47.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.410+0000 7f24574f5780 1 < completed multipart upload key=1V39QA10M1HY upload=XM2N2LT3Z267 tag=SJYM8552K5YM type=write size=1827437 2023-12-03T11:48:47.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.411+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=QWZPFH1QZH7X type=write size=784121 2023-12-03T11:48:47.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.413+0000 7f24574f5780 1 < completed write operation key=1V39QA10M1HY tag=7U01UIV8FLFZ type=write size=682248 2023-12-03T11:48:47.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.415+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.0 size=1048576 2023-12-03T11:48:47.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.416+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.1 size=1048576 2023-12-03T11:48:47.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.417+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.2 size=165659 2023-12-03T11:48:47.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.417+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=0R485DYSXMKP tag=1TJYAWJTCI2D type=write size=3311387 2023-12-03T11:48:47.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.419+0000 7f24574f5780 1 < suggested update operation key=QVLPCWC4T8N8 tag=R2OFMM9FKUL1 type=del size=2041960 2023-12-03T11:48:47.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.421+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~RYXLBPP5Q8VV.0 size=1048576 2023-12-03T11:48:47.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.422+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~RYXLBPP5Q8VV.1 size=714646 2023-12-03T11:48:47.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.422+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=RYXLBPP5Q8VV tag=IK7ZEU20V2AM type=write size=2811798 2023-12-03T11:48:47.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.424+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=GKHGFECQ5VO8 type=del size=3084543 2023-12-03T11:48:47.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.426+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.0 size=1048576 2023-12-03T11:48:47.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.427+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.1 size=1048576 2023-12-03T11:48:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.428+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.2 size=680680 2023-12-03T11:48:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.428+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=52SGHQ41T8X9 tag=165P5YENYVMT type=write size=3826408 2023-12-03T11:48:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.429+0000 7f24574f5780 1 < completed delete operation key=U043YAPGUBT5 tag=05VDCU488P71 type=del 2023-12-03T11:48:47.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.431+0000 7f24574f5780 1 > prepared part key=_multipart_XRHVYVADRWET.2~PHCMJVRO2GHB.0 size=74684 2023-12-03T11:48:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.431+0000 7f24574f5780 1 > prepared multipart upload key=XRHVYVADRWET upload=PHCMJVRO2GHB tag=HI9Y07R0TQIL type=write size=1123260 2023-12-03T11:48:47.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.434+0000 7f24574f5780 1 < suggested remove operation key=SUBILSL92D96 tag=4IOPWNXDZ3E2 type=del 2023-12-03T11:48:47.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.436+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.0 size=1048576 2023-12-03T11:48:47.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.437+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.1 size=1048576 2023-12-03T11:48:47.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.437+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.2 size=2496 2023-12-03T11:48:47.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.437+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=IPOKPKWH8I6F tag=8P9QX9FI2XQK type=write size=3148224 2023-12-03T11:48:47.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.439+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~HNPYFDP7UWCZ.0 size=895902 2023-12-03T11:48:47.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.441+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=HNPYFDP7UWCZ tag=009NNA1EYKAG type=write size=1944478 2023-12-03T11:48:47.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.442+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=BU4KT1XT0J88 type=del size=3816498 2023-12-03T11:48:47.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.444+0000 7f24574f5780 1 < suggested remove operation key=1V1C59DP8YPF tag=Q1Y4RTS9JX5F type=del 2023-12-03T11:48:47.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.445+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=1AMB0W6KKGH2 type=del size=1041338 2023-12-03T11:48:47.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.446+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~PVQB0Z90J604.0 size=413848 2023-12-03T11:48:47.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.448+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=PVQB0Z90J604 tag=09OQJ3L7VEAP type=write size=1462424 2023-12-03T11:48:47.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.449+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=J1PMROB2MHT0 type=write size=162751 2023-12-03T11:48:47.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.450+0000 7f24574f5780 1 < completed write operation key=PGTGVLK4DBN0 tag=EDETX3ETB2QO type=write size=923774 2023-12-03T11:48:47.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.453+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~TMXVZBPO6XN8.0 size=489014 2023-12-03T11:48:47.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.453+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=TMXVZBPO6XN8 tag=N08NCCUTZ0GX type=write size=1537590 2023-12-03T11:48:47.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.455+0000 7f24574f5780 1 < suggested remove operation key=LS4OEAGZWQXC tag=7IN9BH8Y1ATR type=del 2023-12-03T11:48:47.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.456+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=7GB6OJL4VQV0 type=del size=3112653 2023-12-03T11:48:47.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.457+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.0 size=1048576 2023-12-03T11:48:47.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.458+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.1 size=1048576 2023-12-03T11:48:47.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.459+0000 7f24574f5780 1 < completed part key=_multipart_MH1SRAATMNXB.2~TF4QRBGE3KDM.2 size=302116 2023-12-03T11:48:47.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.461+0000 7f24574f5780 1 < completed multipart upload key=MH1SRAATMNXB upload=TF4QRBGE3KDM tag=BE1Y9SF9O4QT type=write size=3447844 2023-12-03T11:48:47.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.462+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=CF7LOP7XR42Y type=del size=1415821 2023-12-03T11:48:47.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.466+0000 7f24574f5780 1 < suggested remove operation key=QVLPCWC4T8N8 tag=1622EYNCAB8H type=del 2023-12-03T11:48:47.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.468+0000 7f24574f5780 1 > prepared operation key=9LKFY1U0JRBQ tag=Z18SRL7MTNNR type=del size=700236 2023-12-03T11:48:47.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.470+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.0 size=1048576 2023-12-03T11:48:47.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.471+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.1 size=1048576 2023-12-03T11:48:47.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.472+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~PS07QBT6WE8Z.2 size=667189 2023-12-03T11:48:47.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.474+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=PS07QBT6WE8Z tag=AUBFIIBCMV9I type=write size=3812917 2023-12-03T11:48:47.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.476+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~ARBKP3Q986AS.0 size=477276 2023-12-03T11:48:47.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.476+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=ARBKP3Q986AS tag=99RFNR8PLDEK type=write size=1525852 2023-12-03T11:48:47.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.477+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~JZIP1NYQWPXF.0 size=1048576 2023-12-03T11:48:47.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.477+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~JZIP1NYQWPXF.1 size=156717 2023-12-03T11:48:47.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.479+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=JZIP1NYQWPXF tag=JLD25AUE1VQR type=write size=2253869 2023-12-03T11:48:47.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.480+0000 7f24574f5780 1 > prepared operation key=PS1OYWI44L80 tag=0C2ZCFXKRSCX type=del size=875042 2023-12-03T11:48:47.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.482+0000 7f24574f5780 1 < completed delete operation key=WZMWCSAZX8RB tag=VPZD4V7EW4DQ type=del 2023-12-03T11:48:47.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.483+0000 7f24574f5780 1 > prepared operation key=QVLPCWC4T8N8 tag=3U3P6SBMI3YH type=del size=3768165 2023-12-03T11:48:47.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.484+0000 7f24574f5780 1 < completed write operation key=DA6ZXCLKTC1L tag=QWZPFH1QZH7X type=write size=784121 2023-12-03T11:48:47.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.486+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.0 size=1048576 2023-12-03T11:48:47.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.487+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.1 size=1048576 2023-12-03T11:48:47.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.488+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.2 size=979267 2023-12-03T11:48:47.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.488+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=P00VU4CVYQA7 tag=Z90ACVKUCCIW type=write size=4124995 2023-12-03T11:48:47.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.489+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.0 size=1048576 2023-12-03T11:48:47.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.489+0000 7f24574f5780 1 < canceled part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.1 size=1048576 2023-12-03T11:48:47.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.490+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~0R485DYSXMKP.2 size=165659 2023-12-03T11:48:47.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.493+0000 7f24574f5780 1 < canceled multipart upload key=BMKG19L35HLV upload=0R485DYSXMKP tag=1TJYAWJTCI2D type=write size=3311387 2023-12-03T11:48:47.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.494+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=MRHTPDW7EBJG type=del size=3209219 2023-12-03T11:48:47.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.495+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~RYXLBPP5Q8VV.0 size=1048576 2023-12-03T11:48:47.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.496+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~RYXLBPP5Q8VV.1 size=714646 2023-12-03T11:48:47.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.498+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=RYXLBPP5Q8VV tag=IK7ZEU20V2AM type=write size=2811798 2023-12-03T11:48:47.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.500+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.0 size=1048576 2023-12-03T11:48:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.501+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.1 size=1048576 2023-12-03T11:48:47.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.502+0000 7f24574f5780 1 > prepared part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.2 size=108798 2023-12-03T11:48:47.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.502+0000 7f24574f5780 1 > prepared multipart upload key=S73M6UFMX5NH upload=HLRXSMIGD9PQ tag=3O7ASKOJ1QQL type=write size=3254526 2023-12-03T11:48:47.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.503+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.0 size=1048576 2023-12-03T11:48:47.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.505+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.1 size=1048576 2023-12-03T11:48:47.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.506+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~52SGHQ41T8X9.2 size=680680 2023-12-03T11:48:47.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.508+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=52SGHQ41T8X9 tag=165P5YENYVMT type=write size=3826408 2023-12-03T11:48:47.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.510+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~2A2LA3RJ0XJV.0 size=1048576 2023-12-03T11:48:47.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.511+0000 7f24574f5780 1 > prepared part key=_multipart_PS1OYWI44L80.2~2A2LA3RJ0XJV.1 size=976363 2023-12-03T11:48:47.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.511+0000 7f24574f5780 1 > prepared multipart upload key=PS1OYWI44L80 upload=2A2LA3RJ0XJV tag=XRH1YLMDMAUS type=write size=3073515 2023-12-03T11:48:47.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.512+0000 7f24574f5780 1 < completed part key=_multipart_XRHVYVADRWET.2~PHCMJVRO2GHB.0 size=74684 2023-12-03T11:48:47.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.514+0000 7f24574f5780 1 < completed multipart upload key=XRHVYVADRWET upload=PHCMJVRO2GHB tag=HI9Y07R0TQIL type=write size=1123260 2023-12-03T11:48:47.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.516+0000 7f24574f5780 1 > prepared part key=_multipart_1QR0TE9DXU5Z.2~D0K4CTARTDD6.0 size=642824 2023-12-03T11:48:47.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.516+0000 7f24574f5780 1 > prepared multipart upload key=1QR0TE9DXU5Z upload=D0K4CTARTDD6 tag=1FRULDZ98U7H type=write size=1691400 2023-12-03T11:48:47.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.517+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.0 size=1048576 2023-12-03T11:48:47.519 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.518+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.1 size=1048576 2023-12-03T11:48:47.520 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.519+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~IPOKPKWH8I6F.2 size=2496 2023-12-03T11:48:47.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.521+0000 7f24574f5780 1 < completed multipart upload key=S7Q5TCM499H1 upload=IPOKPKWH8I6F tag=8P9QX9FI2XQK type=write size=3148224 2023-12-03T11:48:47.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.522+0000 7f24574f5780 1 > prepared operation key=S73M6UFMX5NH tag=9XMRTCV4YSYJ type=del size=129313 2023-12-03T11:48:47.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.524+0000 7f24574f5780 1 < suggested update operation key=WD7U31FW3NGZ tag=BU4KT1XT0J88 type=del size=3816498 2023-12-03T11:48:47.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.526+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2YS5YOLMJQTV.0 size=1048576 2023-12-03T11:48:47.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.527+0000 7f24574f5780 1 > prepared part key=_multipart_WD7U31FW3NGZ.2~2YS5YOLMJQTV.1 size=327295 2023-12-03T11:48:47.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.527+0000 7f24574f5780 1 > prepared multipart upload key=WD7U31FW3NGZ upload=2YS5YOLMJQTV tag=77VUP2DTSFSR type=write size=2424447 2023-12-03T11:48:47.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.528+0000 7f24574f5780 1 < canceled operation key=SC8UOVLQQ0VD tag=1AMB0W6KKGH2 type=del size=1041338 2023-12-03T11:48:47.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.529+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=1T1WJA25WV1A type=del size=3820261 2023-12-03T11:48:47.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.531+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=J1PMROB2MHT0 type=write 2023-12-03T11:48:47.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.532+0000 7f24574f5780 1 > prepared operation key=WD7U31FW3NGZ tag=T2208B7AO7B9 type=del size=3443362 2023-12-03T11:48:47.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.534+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~TMXVZBPO6XN8.0 size=489014 2023-12-03T11:48:47.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.536+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=TMXVZBPO6XN8 tag=N08NCCUTZ0GX type=write size=1537590 2023-12-03T11:48:47.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.537+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=I4Z96F8BRDWU type=del size=1446078 2023-12-03T11:48:47.540 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.539+0000 7f24574f5780 1 < suggested remove operation key=A4LOQE7VRDN7 tag=7GB6OJL4VQV0 type=del 2023-12-03T11:48:47.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.540+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=K7S0OSRXDY42 type=del size=248954 2023-12-03T11:48:47.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.542+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=CF7LOP7XR42Y type=del 2023-12-03T11:48:47.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.544+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~DNWKYKJUIGX1.0 size=1048576 2023-12-03T11:48:47.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.545+0000 7f24574f5780 1 > prepared part key=_multipart_LS4OEAGZWQXC.2~DNWKYKJUIGX1.1 size=586650 2023-12-03T11:48:47.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.545+0000 7f24574f5780 1 > prepared multipart upload key=LS4OEAGZWQXC upload=DNWKYKJUIGX1 tag=G3K4MSEIWJ78 type=write size=2683802 2023-12-03T11:48:47.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.547+0000 7f24574f5780 1 < suggested remove operation key=9LKFY1U0JRBQ tag=Z18SRL7MTNNR type=del 2023-12-03T11:48:47.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.549+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.0 size=1048576 2023-12-03T11:48:47.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.550+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.1 size=1048576 2023-12-03T11:48:47.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.551+0000 7f24574f5780 1 > prepared part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.2 size=73148 2023-12-03T11:48:47.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.551+0000 7f24574f5780 1 > prepared multipart upload key=BMKG19L35HLV upload=6ORVJLNXNBHZ tag=8Y35J3DGW5RT type=write size=3218876 2023-12-03T11:48:47.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.552+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~ARBKP3Q986AS.0 size=477276 2023-12-03T11:48:47.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.554+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=ARBKP3Q986AS tag=99RFNR8PLDEK type=write size=1525852 2023-12-03T11:48:47.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.556+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=T33PN7E9T7AC type=del size=899962 2023-12-03T11:48:47.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.558+0000 7f24574f5780 1 < suggested update operation key=PS1OYWI44L80 tag=0C2ZCFXKRSCX type=del size=875042 2023-12-03T11:48:47.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.559+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=T23HWXBAY5AD type=del size=2623511 2023-12-03T11:48:47.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.562+0000 7f24574f5780 1 < suggested update operation key=QVLPCWC4T8N8 tag=3U3P6SBMI3YH type=del size=3768165 2023-12-03T11:48:47.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.563+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=R460Y8XA8L9T type=del size=3424619 2023-12-03T11:48:47.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.564+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.0 size=1048576 2023-12-03T11:48:47.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.564+0000 7f24574f5780 1 < canceled part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.1 size=1048576 2023-12-03T11:48:47.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.565+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~P00VU4CVYQA7.2 size=979267 2023-12-03T11:48:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.567+0000 7f24574f5780 1 < canceled multipart upload key=A4LOQE7VRDN7 upload=P00VU4CVYQA7 tag=Z90ACVKUCCIW type=write size=4124995 2023-12-03T11:48:47.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.568+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=1DUDY2DWOAVJ type=del size=1711122 2023-12-03T11:48:47.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.569+0000 7f24574f5780 -1 < no bi entry to suggest for operation key=A4LOQE7VRDN7 tag=MRHTPDW7EBJG type=del size=3209219: (2) No such file or directory 2023-12-03T11:48:47.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.570+0000 7f24574f5780 1 > prepared operation key=DA6ZXCLKTC1L tag=C30BOR9W1URB type=del size=3420682 2023-12-03T11:48:47.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.571+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.0 size=1048576 2023-12-03T11:48:47.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.572+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.1 size=1048576 2023-12-03T11:48:47.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.573+0000 7f24574f5780 1 < completed part key=_multipart_S73M6UFMX5NH.2~HLRXSMIGD9PQ.2 size=108798 2023-12-03T11:48:47.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.575+0000 7f24574f5780 1 < completed multipart upload key=S73M6UFMX5NH upload=HLRXSMIGD9PQ tag=3O7ASKOJ1QQL type=write size=3254526 2023-12-03T11:48:47.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.578+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~B0YE0L9QFIEI.0 size=1048576 2023-12-03T11:48:47.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.579+0000 7f24574f5780 1 > prepared part key=_multipart_PGTGVLK4DBN0.2~B0YE0L9QFIEI.1 size=881098 2023-12-03T11:48:47.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.579+0000 7f24574f5780 1 > prepared multipart upload key=PGTGVLK4DBN0 upload=B0YE0L9QFIEI tag=7083MTHQPZDE type=write size=2978250 2023-12-03T11:48:47.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.581+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~2A2LA3RJ0XJV.0 size=1048576 2023-12-03T11:48:47.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.581+0000 7f24574f5780 1 < completed part key=_multipart_PS1OYWI44L80.2~2A2LA3RJ0XJV.1 size=976363 2023-12-03T11:48:47.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.583+0000 7f24574f5780 1 < completed multipart upload key=PS1OYWI44L80 upload=2A2LA3RJ0XJV tag=XRH1YLMDMAUS type=write size=3073515 2023-12-03T11:48:47.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.585+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.0 size=1048576 2023-12-03T11:48:47.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.586+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.1 size=1048576 2023-12-03T11:48:47.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.587+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.2 size=975049 2023-12-03T11:48:47.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.587+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=NCHD9BTQJZ8B tag=EMMWQJS1IRUW type=write size=4120777 2023-12-03T11:48:47.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.589+0000 7f24574f5780 1 < completed part key=_multipart_1QR0TE9DXU5Z.2~D0K4CTARTDD6.0 size=642824 2023-12-03T11:48:47.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.591+0000 7f24574f5780 1 < completed multipart upload key=1QR0TE9DXU5Z upload=D0K4CTARTDD6 tag=1FRULDZ98U7H type=write size=1691400 2023-12-03T11:48:47.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.593+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~8C8W2D1TJ381.0 size=1048576 2023-12-03T11:48:47.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.594+0000 7f24574f5780 1 > prepared part key=_multipart_A4LOQE7VRDN7.2~8C8W2D1TJ381.1 size=1036005 2023-12-03T11:48:47.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.594+0000 7f24574f5780 1 > prepared multipart upload key=A4LOQE7VRDN7 upload=8C8W2D1TJ381 tag=YV1SLIMKOG1T type=write size=3133157 2023-12-03T11:48:47.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.595+0000 7f24574f5780 1 < completed delete operation key=S73M6UFMX5NH tag=9XMRTCV4YSYJ type=del 2023-12-03T11:48:47.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.596+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=TBT02VXRBBI6 type=del size=3286487 2023-12-03T11:48:47.600 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.599+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2YS5YOLMJQTV.0 size=1048576 2023-12-03T11:48:47.601 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.600+0000 7f24574f5780 1 < completed part key=_multipart_WD7U31FW3NGZ.2~2YS5YOLMJQTV.1 size=327295 2023-12-03T11:48:47.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.602+0000 7f24574f5780 1 < completed multipart upload key=WD7U31FW3NGZ upload=2YS5YOLMJQTV tag=77VUP2DTSFSR type=write size=2424447 2023-12-03T11:48:47.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.604+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=GE2C1FKN98DU type=del size=2081446 2023-12-03T11:48:47.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.606+0000 7f24574f5780 1 < suggested remove operation key=ON2E6H35N9FJ tag=1T1WJA25WV1A type=del 2023-12-03T11:48:47.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.607+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=AOBXTVE4KWHY type=del size=3017853 2023-12-03T11:48:47.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.609+0000 7f24574f5780 1 < suggested remove operation key=WD7U31FW3NGZ tag=T2208B7AO7B9 type=del 2023-12-03T11:48:47.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.611+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=79Q0CF6L5UEX type=del size=1987851 2023-12-03T11:48:47.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.612+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=I4Z96F8BRDWU type=del 2023-12-03T11:48:47.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.613+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=6UJFSUJTL8AG type=del size=1081575 2023-12-03T11:48:47.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.614+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=K7S0OSRXDY42 type=del 2023-12-03T11:48:47.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.616+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~OG83I8QXYSB2.0 size=1048576 2023-12-03T11:48:47.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.617+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~OG83I8QXYSB2.1 size=315963 2023-12-03T11:48:47.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.617+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=OG83I8QXYSB2 tag=BC4FLNYNX1MW type=write size=2413115 2023-12-03T11:48:47.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.618+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~DNWKYKJUIGX1.0 size=1048576 2023-12-03T11:48:47.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.621+0000 7f24574f5780 1 < completed part key=_multipart_LS4OEAGZWQXC.2~DNWKYKJUIGX1.1 size=586650 2023-12-03T11:48:47.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.623+0000 7f24574f5780 1 < completed multipart upload key=LS4OEAGZWQXC upload=DNWKYKJUIGX1 tag=G3K4MSEIWJ78 type=write size=2683802 2023-12-03T11:48:47.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.624+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=USH64QIDH7QK type=write size=956400 2023-12-03T11:48:47.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.625+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.0 size=1048576 2023-12-03T11:48:47.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.626+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.1 size=1048576 2023-12-03T11:48:47.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.627+0000 7f24574f5780 1 < completed part key=_multipart_BMKG19L35HLV.2~6ORVJLNXNBHZ.2 size=73148 2023-12-03T11:48:47.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.629+0000 7f24574f5780 1 < completed multipart upload key=BMKG19L35HLV upload=6ORVJLNXNBHZ tag=8Y35J3DGW5RT type=write size=3218876 2023-12-03T11:48:47.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.630+0000 7f24574f5780 1 > prepared operation key=1V1C59DP8YPF tag=057ORBN4U5PS type=del size=3777112 2023-12-03T11:48:47.633 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.632+0000 7f24574f5780 -1 < failed to complete operation key=ON2E6H35N9FJ tag=T33PN7E9T7AC type=del size=899962: (22) Invalid argument 2023-12-03T11:48:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.633+0000 7f24574f5780 1 > prepared operation key=XZFHFS76WNR4 tag=X4QOR76QD69Z type=del size=506268 2023-12-03T11:48:47.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.634+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=T23HWXBAY5AD type=del 2023-12-03T11:48:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.635+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=IFN11UVYXAFW type=del size=4059122 2023-12-03T11:48:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.636+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=R460Y8XA8L9T type=del 2023-12-03T11:48:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.638+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=XRGUSFBOOFRS type=del size=145833 2023-12-03T11:48:47.641 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.640+0000 7f24574f5780 1 < suggested remove operation key=DA6ZXCLKTC1L tag=1DUDY2DWOAVJ type=del 2023-12-03T11:48:47.642 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.641+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=KZWYYIY7Y2HK type=del size=3689626 2023-12-03T11:48:47.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.642+0000 7f24574f5780 -1 < failed to complete operation key=DA6ZXCLKTC1L tag=C30BOR9W1URB type=del size=3420682: (22) Invalid argument 2023-12-03T11:48:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.644+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~UAVA7ETKCTP0.0 size=315427 2023-12-03T11:48:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.644+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=UAVA7ETKCTP0 tag=8VEWM34UWYLK type=write size=1364003 2023-12-03T11:48:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.645+0000 7f24574f5780 1 < canceled part key=_multipart_PGTGVLK4DBN0.2~B0YE0L9QFIEI.0 size=1048576 2023-12-03T11:48:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.646+0000 7f24574f5780 1 < completed part key=_multipart_PGTGVLK4DBN0.2~B0YE0L9QFIEI.1 size=881098 2023-12-03T11:48:47.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.648+0000 7f24574f5780 1 < canceled multipart upload key=PGTGVLK4DBN0 upload=B0YE0L9QFIEI tag=7083MTHQPZDE type=write size=2978250 2023-12-03T11:48:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.650+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.0 size=1048576 2023-12-03T11:48:47.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.651+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.1 size=1048576 2023-12-03T11:48:47.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.652+0000 7f24574f5780 1 > prepared part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.2 size=243499 2023-12-03T11:48:47.653 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.652+0000 7f24574f5780 1 > prepared multipart upload key=S7Q5TCM499H1 upload=13WA5BDDZ7RV tag=4J1E1G2TS3W6 type=write size=3389227 2023-12-03T11:48:47.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.653+0000 7f24574f5780 1 < completed part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.0 size=1048576 2023-12-03T11:48:47.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.653+0000 7f24574f5780 1 < canceled part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.1 size=1048576 2023-12-03T11:48:47.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.653+0000 7f24574f5780 1 < canceled part key=_multipart_SUBILSL92D96.2~NCHD9BTQJZ8B.2 size=975049 2023-12-03T11:48:47.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.655+0000 7f24574f5780 1 < canceled multipart upload key=SUBILSL92D96 upload=NCHD9BTQJZ8B tag=EMMWQJS1IRUW type=write size=4120777 2023-12-03T11:48:47.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.657+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~MCQDEYW8IKHC.0 size=1006256 2023-12-03T11:48:47.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.657+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=MCQDEYW8IKHC tag=N9NJIIFMPXCK type=write size=2054832 2023-12-03T11:48:47.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.658+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~8C8W2D1TJ381.0 size=1048576 2023-12-03T11:48:47.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.659+0000 7f24574f5780 1 < completed part key=_multipart_A4LOQE7VRDN7.2~8C8W2D1TJ381.1 size=1036005 2023-12-03T11:48:47.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.661+0000 7f24574f5780 1 < completed multipart upload key=A4LOQE7VRDN7 upload=8C8W2D1TJ381 tag=YV1SLIMKOG1T type=write size=3133157 2023-12-03T11:48:47.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.663+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.0 size=1048576 2023-12-03T11:48:47.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.664+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.1 size=1048576 2023-12-03T11:48:47.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.665+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.2 size=275680 2023-12-03T11:48:47.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.665+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=IX3VS0H7AL9J tag=NREE4PCJ6532 type=write size=3421408 2023-12-03T11:48:47.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.667+0000 7f24574f5780 1 < suggested update operation key=U043YAPGUBT5 tag=TBT02VXRBBI6 type=del size=3286487 2023-12-03T11:48:47.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.668+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=D2SM42RO1XPS type=del size=624104 2023-12-03T11:48:47.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.670+0000 7f24574f5780 1 < completed delete operation key=1V1C59DP8YPF tag=GE2C1FKN98DU type=del 2023-12-03T11:48:47.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.671+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=OF8XQ3JOAWH4 type=del size=3323753 2023-12-03T11:48:47.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.672+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=AOBXTVE4KWHY type=del 2023-12-03T11:48:47.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.673+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=V9EK8WA1DSX4 type=del size=3203982 2023-12-03T11:48:47.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.674+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=79Q0CF6L5UEX type=del 2023-12-03T11:48:47.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.675+0000 7f24574f5780 1 > prepared operation key=WZMWCSAZX8RB tag=F5EDE8UBL8VG type=write size=510067 2023-12-03T11:48:47.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.677+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=6UJFSUJTL8AG type=del 2023-12-03T11:48:47.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.679+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.0 size=1048576 2023-12-03T11:48:47.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.680+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.1 size=1048576 2023-12-03T11:48:47.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.681+0000 7f24574f5780 1 > prepared part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.2 size=616446 2023-12-03T11:48:47.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.681+0000 7f24574f5780 1 > prepared multipart upload key=XO42YZR60KQE upload=PWL804WPYYES tag=SMDHGVYD9H84 type=write size=3762174 2023-12-03T11:48:47.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.682+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~OG83I8QXYSB2.0 size=1048576 2023-12-03T11:48:47.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.683+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~OG83I8QXYSB2.1 size=315963 2023-12-03T11:48:47.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.687+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=OG83I8QXYSB2 tag=BC4FLNYNX1MW type=write size=2413115 2023-12-03T11:48:47.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.688+0000 7f24574f5780 1 > prepared operation key=SUBILSL92D96 tag=1EPWBKNZFXEY type=del size=3382094 2023-12-03T11:48:47.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.689+0000 7f24574f5780 1 < completed write operation key=ON2E6H35N9FJ tag=USH64QIDH7QK type=write size=956400 2023-12-03T11:48:47.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.690+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=D2451MRL6IZC type=del size=3821197 2023-12-03T11:48:47.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.693+0000 7f24574f5780 1 < canceled operation key=1V1C59DP8YPF tag=057ORBN4U5PS type=del size=3777112 2023-12-03T11:48:47.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.694+0000 7f24574f5780 1 > prepared operation key=BMKG19L35HLV tag=LEQD1YAUJOLC type=del size=2357653 2023-12-03T11:48:47.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.695+0000 7f24574f5780 1 < completed delete operation key=XZFHFS76WNR4 tag=X4QOR76QD69Z type=del 2023-12-03T11:48:47.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.696+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=RDYUWU9NDGJJ type=del size=2805277 2023-12-03T11:48:47.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.697+0000 7f24574f5780 1 < canceled operation key=S7Q5TCM499H1 tag=IFN11UVYXAFW type=del size=4059122 2023-12-03T11:48:47.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.699+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.0 size=1048576 2023-12-03T11:48:47.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.700+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.1 size=1048576 2023-12-03T11:48:47.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.701+0000 7f24574f5780 1 > prepared part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.2 size=780433 2023-12-03T11:48:47.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.701+0000 7f24574f5780 1 > prepared multipart upload key=ON2E6H35N9FJ upload=4TS6WYOOMFQY tag=9SCEWRFL2CEQ type=write size=3926161 2023-12-03T11:48:47.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.702+0000 7f24574f5780 1 < completed delete operation key=863EOZDYRWH3 tag=XRGUSFBOOFRS type=del 2023-12-03T11:48:47.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.704+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=39AU3O1QF8NR type=del size=2281947 2023-12-03T11:48:47.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.706+0000 7f24574f5780 1 < suggested update operation key=3EOEF9UMATIE tag=KZWYYIY7Y2HK type=del size=3689626 2023-12-03T11:48:47.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.708+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=TY5UX04XEGZT type=write size=894658 2023-12-03T11:48:47.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.709+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~UAVA7ETKCTP0.0 size=315427 2023-12-03T11:48:47.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.711+0000 7f24574f5780 1 < completed multipart upload key=1V1C59DP8YPF upload=UAVA7ETKCTP0 tag=8VEWM34UWYLK type=write size=1364003 2023-12-03T11:48:47.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.712+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=73UGQMZF0P3S type=write size=468001 2023-12-03T11:48:47.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.713+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.0 size=1048576 2023-12-03T11:48:47.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.714+0000 7f24574f5780 1 < completed part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.1 size=1048576 2023-12-03T11:48:47.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.714+0000 7f24574f5780 1 < canceled part key=_multipart_S7Q5TCM499H1.2~13WA5BDDZ7RV.2 size=243499 2023-12-03T11:48:47.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.716+0000 7f24574f5780 1 < canceled multipart upload key=S7Q5TCM499H1 upload=13WA5BDDZ7RV tag=4J1E1G2TS3W6 type=write size=3389227 2023-12-03T11:48:47.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.718+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=MSUF85RWZE1N type=del size=3005014 2023-12-03T11:48:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.719+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~MCQDEYW8IKHC.0 size=1006256 2023-12-03T11:48:47.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.721+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=MCQDEYW8IKHC tag=N9NJIIFMPXCK type=write size=2054832 2023-12-03T11:48:47.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.723+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~48ZB1JP30PU0.0 size=1048576 2023-12-03T11:48:47.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.723+0000 7f24574f5780 1 > prepared part key=_multipart_1V39QA10M1HY.2~48ZB1JP30PU0.1 size=324543 2023-12-03T11:48:47.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.723+0000 7f24574f5780 1 > prepared multipart upload key=1V39QA10M1HY upload=48ZB1JP30PU0 tag=QCVXMMI255KL type=write size=2421695 2023-12-03T11:48:47.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.724+0000 7f24574f5780 1 < canceled part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.0 size=1048576 2023-12-03T11:48:47.726 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.725+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.1 size=1048576 2023-12-03T11:48:47.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.725+0000 7f24574f5780 1 < completed part key=_multipart_1V1C59DP8YPF.2~IX3VS0H7AL9J.2 size=275680 2023-12-03T11:48:47.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.728+0000 7f24574f5780 1 < canceled multipart upload key=1V1C59DP8YPF upload=IX3VS0H7AL9J tag=NREE4PCJ6532 type=write size=3421408 2023-12-03T11:48:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.730+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.0 size=1048576 2023-12-03T11:48:47.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.731+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.1 size=1048576 2023-12-03T11:48:47.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.731+0000 7f24574f5780 1 > prepared part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.2 size=572446 2023-12-03T11:48:47.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.731+0000 7f24574f5780 1 > prepared multipart upload key=BV8C0VSJ1GTW upload=IG9QYPX4HKBA tag=HCD7B29B7IRC type=write size=3718174 2023-12-03T11:48:47.734 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.733+0000 7f24574f5780 1 < completed delete operation key=S0EH0Q0UA8HG tag=D2SM42RO1XPS type=del 2023-12-03T11:48:47.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.735+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.0 size=1048576 2023-12-03T11:48:47.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.736+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.1 size=1048576 2023-12-03T11:48:47.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.737+0000 7f24574f5780 1 > prepared part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.2 size=137566 2023-12-03T11:48:47.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.737+0000 7f24574f5780 1 > prepared multipart upload key=WZMWCSAZX8RB upload=3DCN7K9LONKI tag=6MNZPM3J6EUH type=write size=3283294 2023-12-03T11:48:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.738+0000 7f24574f5780 1 < completed delete operation key=XO42YZR60KQE tag=OF8XQ3JOAWH4 type=del 2023-12-03T11:48:47.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.739+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=FPOVEGJEMCC5 type=write size=64607 2023-12-03T11:48:47.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.740+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=V9EK8WA1DSX4 type=del 2023-12-03T11:48:47.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.741+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=2ANDQ60UDA57 type=write size=681102 2023-12-03T11:48:47.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.743+0000 7f24574f5780 1 < completed write operation key=WZMWCSAZX8RB tag=F5EDE8UBL8VG type=write size=510067 2023-12-03T11:48:47.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.744+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=A2QSAO4I94ZS type=del size=1548068 2023-12-03T11:48:47.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.745+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.0 size=1048576 2023-12-03T11:48:47.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.746+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.1 size=1048576 2023-12-03T11:48:47.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.747+0000 7f24574f5780 1 < completed part key=_multipart_XO42YZR60KQE.2~PWL804WPYYES.2 size=616446 2023-12-03T11:48:47.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.749+0000 7f24574f5780 1 < completed multipart upload key=XO42YZR60KQE upload=PWL804WPYYES tag=SMDHGVYD9H84 type=write size=3762174 2023-12-03T11:48:47.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.750+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=ZV7B8VZB693S type=del size=2034407 2023-12-03T11:48:47.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.753+0000 7f24574f5780 1 < suggested update operation key=SUBILSL92D96 tag=1EPWBKNZFXEY type=del size=3382094 2023-12-03T11:48:47.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.755+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.0 size=1048576 2023-12-03T11:48:47.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.756+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.1 size=1048576 2023-12-03T11:48:47.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.757+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.2 size=557038 2023-12-03T11:48:47.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.757+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=G32AR8H3AECM tag=S512AVZB0P5K type=write size=3702766 2023-12-03T11:48:47.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.758+0000 7f24574f5780 1 < completed delete operation key=S7Q5TCM499H1 tag=D2451MRL6IZC type=del 2023-12-03T11:48:47.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.759+0000 7f24574f5780 1 > prepared operation key=35U7YCDYMX46 tag=20A3YM3A3GQD type=del size=3748666 2023-12-03T11:48:47.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.760+0000 7f24574f5780 1 < completed delete operation key=BMKG19L35HLV tag=LEQD1YAUJOLC type=del 2023-12-03T11:48:47.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.761+0000 7f24574f5780 1 > prepared operation key=863EOZDYRWH3 tag=504HEMX6ISCB type=write size=120133 2023-12-03T11:48:47.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.762+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=RDYUWU9NDGJJ type=del 2023-12-03T11:48:47.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.764+0000 7f24574f5780 1 > prepared operation key=SC8UOVLQQ0VD tag=3OPPRVUI556S type=del size=3273931 2023-12-03T11:48:47.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.765+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.0 size=1048576 2023-12-03T11:48:47.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.766+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.1 size=1048576 2023-12-03T11:48:47.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.767+0000 7f24574f5780 1 < completed part key=_multipart_ON2E6H35N9FJ.2~4TS6WYOOMFQY.2 size=780433 2023-12-03T11:48:47.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.769+0000 7f24574f5780 1 < completed multipart upload key=ON2E6H35N9FJ upload=4TS6WYOOMFQY tag=9SCEWRFL2CEQ type=write size=3926161 2023-12-03T11:48:47.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.770+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=0PU5WIWMD5AO type=del size=421456 2023-12-03T11:48:47.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.771+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=39AU3O1QF8NR type=del 2023-12-03T11:48:47.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.773+0000 7f24574f5780 1 > prepared operation key=XRHVYVADRWET tag=KMJZLKI73TXN type=write size=204421 2023-12-03T11:48:47.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.774+0000 7f24574f5780 1 < completed write operation key=A4LOQE7VRDN7 tag=TY5UX04XEGZT type=write size=894658 2023-12-03T11:48:47.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.775+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=6ZEWA6P2YB0C type=del size=3233329 2023-12-03T11:48:47.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.776+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=73UGQMZF0P3S type=write size=468001 2023-12-03T11:48:47.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.778+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~7FZGOTJXRC73.0 size=1048576 2023-12-03T11:48:47.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.779+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~7FZGOTJXRC73.1 size=925516 2023-12-03T11:48:47.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.779+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=7FZGOTJXRC73 tag=XV06OIF9W6MA type=write size=3022668 2023-12-03T11:48:47.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.781+0000 7f24574f5780 1 < completed delete operation key=PGTGVLK4DBN0 tag=MSUF85RWZE1N type=del 2023-12-03T11:48:47.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.782+0000 7f24574f5780 1 > prepared operation key=BV8C0VSJ1GTW tag=2B32GQL2FNNK type=del size=425319 2023-12-03T11:48:47.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.783+0000 7f24574f5780 1 < completed part key=_multipart_1V39QA10M1HY.2~48ZB1JP30PU0.0 size=1048576 2023-12-03T11:48:47.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.783+0000 7f24574f5780 1 < canceled part key=_multipart_1V39QA10M1HY.2~48ZB1JP30PU0.1 size=324543 2023-12-03T11:48:47.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.785+0000 7f24574f5780 1 < canceled multipart upload key=1V39QA10M1HY upload=48ZB1JP30PU0 tag=QCVXMMI255KL type=write size=2421695 2023-12-03T11:48:47.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.786+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=PSOX2ZP15XQU type=del size=1891883 2023-12-03T11:48:47.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.787+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.0 size=1048576 2023-12-03T11:48:47.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.788+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.1 size=1048576 2023-12-03T11:48:47.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.789+0000 7f24574f5780 1 < completed part key=_multipart_BV8C0VSJ1GTW.2~IG9QYPX4HKBA.2 size=572446 2023-12-03T11:48:47.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.791+0000 7f24574f5780 1 < completed multipart upload key=BV8C0VSJ1GTW upload=IG9QYPX4HKBA tag=HCD7B29B7IRC type=write size=3718174 2023-12-03T11:48:47.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.793+0000 7f24574f5780 1 > prepared part key=_multipart_6WY91DFIRUN8.2~ODQMFXBOJLZ5.0 size=238389 2023-12-03T11:48:47.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.793+0000 7f24574f5780 1 > prepared multipart upload key=6WY91DFIRUN8 upload=ODQMFXBOJLZ5 tag=DOA5G6AFUCRX type=write size=1286965 2023-12-03T11:48:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.794+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.0 size=1048576 2023-12-03T11:48:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.795+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.1 size=1048576 2023-12-03T11:48:47.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.796+0000 7f24574f5780 1 < completed part key=_multipart_WZMWCSAZX8RB.2~3DCN7K9LONKI.2 size=137566 2023-12-03T11:48:47.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.798+0000 7f24574f5780 1 < completed multipart upload key=WZMWCSAZX8RB upload=3DCN7K9LONKI tag=6MNZPM3J6EUH type=write size=3283294 2023-12-03T11:48:47.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.799+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=43P0PHXU6V7D type=write size=879810 2023-12-03T11:48:47.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.800+0000 7f24574f5780 1 < completed write operation key=SC8UOVLQQ0VD tag=FPOVEGJEMCC5 type=write size=64607 2023-12-03T11:48:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.804+0000 7f24574f5780 1 > prepared operation key=HM9JHOV8W24F tag=QL30G88UOKJ0 type=del size=711495 2023-12-03T11:48:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.806+0000 7f24574f5780 1 < completed write operation key=U043YAPGUBT5 tag=2ANDQ60UDA57 type=write size=681102 2023-12-03T11:48:47.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.808+0000 7f24574f5780 1 > prepared part key=_multipart_863EOZDYRWH3.2~WN959ANZ4H6M.0 size=1013895 2023-12-03T11:48:47.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.808+0000 7f24574f5780 1 > prepared multipart upload key=863EOZDYRWH3 upload=WN959ANZ4H6M tag=8K5A2R5FM959 type=write size=2062471 2023-12-03T11:48:47.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.809+0000 7f24574f5780 1 < completed delete operation key=ON2E6H35N9FJ tag=A2QSAO4I94ZS type=del 2023-12-03T11:48:47.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.810+0000 7f24574f5780 1 > prepared operation key=S7Q5TCM499H1 tag=PI2XGZSLD8EN type=write size=888882 2023-12-03T11:48:47.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.812+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=ZV7B8VZB693S type=del 2023-12-03T11:48:47.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.813+0000 7f24574f5780 1 > prepared operation key=RABUR02D2ZMD tag=3S1595QBHBP8 type=del size=1960669 2023-12-03T11:48:47.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.814+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.0 size=1048576 2023-12-03T11:48:47.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.815+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.1 size=1048576 2023-12-03T11:48:47.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.815+0000 7f24574f5780 1 < canceled part key=_multipart_DA6ZXCLKTC1L.2~G32AR8H3AECM.2 size=557038 2023-12-03T11:48:47.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.817+0000 7f24574f5780 1 < canceled multipart upload key=DA6ZXCLKTC1L upload=G32AR8H3AECM tag=S512AVZB0P5K type=write size=3702766 2023-12-03T11:48:47.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.818+0000 7f24574f5780 1 > prepared operation key=LS4OEAGZWQXC tag=JQ376ERN7OJO type=del size=2314020 2023-12-03T11:48:47.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.819+0000 7f24574f5780 1 < completed delete operation key=35U7YCDYMX46 tag=20A3YM3A3GQD type=del 2023-12-03T11:48:47.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.821+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.0 size=1048576 2023-12-03T11:48:47.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.822+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.1 size=1048576 2023-12-03T11:48:47.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.823+0000 7f24574f5780 1 > prepared part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.2 size=460175 2023-12-03T11:48:47.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.823+0000 7f24574f5780 1 > prepared multipart upload key=S0EH0Q0UA8HG upload=6BYSBF2AVJPX tag=K21RFYQVU8MX type=write size=3605903 2023-12-03T11:48:47.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.824+0000 7f24574f5780 1 < completed write operation key=863EOZDYRWH3 tag=504HEMX6ISCB type=write size=120133 2023-12-03T11:48:47.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.825+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=1SU07MRHS2AB type=write size=333251 2023-12-03T11:48:47.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.826+0000 7f24574f5780 1 < completed delete operation key=SC8UOVLQQ0VD tag=3OPPRVUI556S type=del 2023-12-03T11:48:47.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.827+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=EXCH1VJVYCA2 type=del size=3347089 2023-12-03T11:48:47.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.828+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=0PU5WIWMD5AO type=del 2023-12-03T11:48:47.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.830+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~FPAW4CUEM18S.0 size=1048576 2023-12-03T11:48:47.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.831+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~FPAW4CUEM18S.1 size=673471 2023-12-03T11:48:47.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.831+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=FPAW4CUEM18S tag=TW931KCEVSW5 type=write size=2770623 2023-12-03T11:48:47.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.832+0000 7f24574f5780 1 < completed write operation key=XRHVYVADRWET tag=KMJZLKI73TXN type=write size=204421 2023-12-03T11:48:47.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.834+0000 7f24574f5780 1 > prepared part key=_multipart_3EOEF9UMATIE.2~FU9SPQXKXU4U.0 size=432902 2023-12-03T11:48:47.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.834+0000 7f24574f5780 1 > prepared multipart upload key=3EOEF9UMATIE upload=FU9SPQXKXU4U tag=3IWMIS14WVWX type=write size=1481478 2023-12-03T11:48:47.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.836+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=6ZEWA6P2YB0C type=del 2023-12-03T11:48:47.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.837+0000 7f24574f5780 1 > prepared operation key=3EOEF9UMATIE tag=AXSDK0UBZQKG type=del size=1597680 2023-12-03T11:48:47.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.838+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~7FZGOTJXRC73.0 size=1048576 2023-12-03T11:48:47.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.839+0000 7f24574f5780 1 < completed part key=_multipart_3EOEF9UMATIE.2~7FZGOTJXRC73.1 size=925516 2023-12-03T11:48:47.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.841+0000 7f24574f5780 1 < completed multipart upload key=3EOEF9UMATIE upload=7FZGOTJXRC73 tag=XV06OIF9W6MA type=write size=3022668 2023-12-03T11:48:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.843+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1RQDSX2ZGLNH.0 size=1048576 2023-12-03T11:48:47.845 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.844+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1RQDSX2ZGLNH.1 size=1048576 2023-12-03T11:48:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.845+0000 7f24574f5780 1 > prepared part key=_multipart_1V1C59DP8YPF.2~1RQDSX2ZGLNH.2 size=936643 2023-12-03T11:48:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.845+0000 7f24574f5780 1 > prepared multipart upload key=1V1C59DP8YPF upload=1RQDSX2ZGLNH tag=4MW2ZJ4S79P5 type=write size=4082371 2023-12-03T11:48:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.846+0000 7f24574f5780 1 < completed delete operation key=BV8C0VSJ1GTW tag=2B32GQL2FNNK type=del 2023-12-03T11:48:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.847+0000 7f24574f5780 1 > prepared operation key=PGTGVLK4DBN0 tag=IRG8OVQ2L886 type=del size=3038984 2023-12-03T11:48:47.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.848+0000 7f24574f5780 1 < completed delete operation key=1QR0TE9DXU5Z tag=PSOX2ZP15XQU type=del 2023-12-03T11:48:47.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.849+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=9C5MOIHDIFTC type=del size=459741 2023-12-03T11:48:47.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.850+0000 7f24574f5780 1 < completed part key=_multipart_6WY91DFIRUN8.2~ODQMFXBOJLZ5.0 size=238389 2023-12-03T11:48:47.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.853+0000 7f24574f5780 1 < completed multipart upload key=6WY91DFIRUN8 upload=ODQMFXBOJLZ5 tag=DOA5G6AFUCRX type=write size=1286965 2023-12-03T11:48:47.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.854+0000 7f24574f5780 1 > prepared operation key=6WY91DFIRUN8 tag=BLGVOEXTDDYQ type=write size=833128 2023-12-03T11:48:47.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.855+0000 7f24574f5780 1 < completed write operation key=LS4OEAGZWQXC tag=43P0PHXU6V7D type=write size=879810 2023-12-03T11:48:47.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.856+0000 7f24574f5780 1 > prepared operation key=ON2E6H35N9FJ tag=ZPI05A4UFM0P type=del size=1433747 2023-12-03T11:48:47.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.857+0000 7f24574f5780 1 < completed delete operation key=HM9JHOV8W24F tag=QL30G88UOKJ0 type=del 2023-12-03T11:48:47.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.860+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~Z9P79FFQ8QKZ.0 size=1048576 2023-12-03T11:48:47.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.861+0000 7f24574f5780 1 > prepared part key=_multipart_XZFHFS76WNR4.2~Z9P79FFQ8QKZ.1 size=397726 2023-12-03T11:48:47.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.861+0000 7f24574f5780 1 > prepared multipart upload key=XZFHFS76WNR4 upload=Z9P79FFQ8QKZ tag=W5D4A80OHT3F type=write size=2494878 2023-12-03T11:48:47.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.862+0000 7f24574f5780 1 < completed part key=_multipart_863EOZDYRWH3.2~WN959ANZ4H6M.0 size=1013895 2023-12-03T11:48:47.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.864+0000 7f24574f5780 1 < completed multipart upload key=863EOZDYRWH3 upload=WN959ANZ4H6M tag=8K5A2R5FM959 type=write size=2062471 2023-12-03T11:48:47.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.865+0000 7f24574f5780 1 > prepared operation key=1QR0TE9DXU5Z tag=WB2SZY2CY5N7 type=del size=3917350 2023-12-03T11:48:47.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.866+0000 7f24574f5780 1 < completed write operation key=S7Q5TCM499H1 tag=PI2XGZSLD8EN type=write size=888882 2023-12-03T11:48:47.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.867+0000 7f24574f5780 1 > prepared operation key=A4LOQE7VRDN7 tag=LHV95QWNW8EI type=write size=924149 2023-12-03T11:48:47.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.868+0000 7f24574f5780 1 < completed delete operation key=RABUR02D2ZMD tag=3S1595QBHBP8 type=del 2023-12-03T11:48:47.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.870+0000 7f24574f5780 1 > prepared operation key=U043YAPGUBT5 tag=NOEBVUVKDI6K type=del size=721481 2023-12-03T11:48:47.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.871+0000 7f24574f5780 1 < completed delete operation key=LS4OEAGZWQXC tag=JQ376ERN7OJO type=del 2023-12-03T11:48:47.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.873+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~DEMHUCUGFHE1.0 size=1048576 2023-12-03T11:48:47.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.874+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~DEMHUCUGFHE1.1 size=1048576 2023-12-03T11:48:47.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.875+0000 7f24574f5780 1 > prepared part key=_multipart_SUBILSL92D96.2~DEMHUCUGFHE1.2 size=149515 2023-12-03T11:48:47.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.875+0000 7f24574f5780 1 > prepared multipart upload key=SUBILSL92D96 upload=DEMHUCUGFHE1 tag=8K2VLF0SUBMM type=write size=3295243 2023-12-03T11:48:47.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.876+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.0 size=1048576 2023-12-03T11:48:47.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.877+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.1 size=1048576 2023-12-03T11:48:47.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.878+0000 7f24574f5780 1 < completed part key=_multipart_S0EH0Q0UA8HG.2~6BYSBF2AVJPX.2 size=460175 2023-12-03T11:48:47.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.880+0000 7f24574f5780 1 < completed multipart upload key=S0EH0Q0UA8HG upload=6BYSBF2AVJPX tag=K21RFYQVU8MX type=write size=3605903 2023-12-03T11:48:47.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.881+0000 7f24574f5780 1 > prepared operation key=XO42YZR60KQE tag=FY5EL1BY2JT6 type=del size=690639 2023-12-03T11:48:47.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.882+0000 7f24574f5780 1 < canceled operation key=ON2E6H35N9FJ tag=1SU07MRHS2AB type=write size=333251 2023-12-03T11:48:47.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.884+0000 7f24574f5780 1 > prepared part key=_multipart_DA6ZXCLKTC1L.2~OKG9XG21MISM.0 size=578085 2023-12-03T11:48:47.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.884+0000 7f24574f5780 1 > prepared multipart upload key=DA6ZXCLKTC1L upload=OKG9XG21MISM tag=Y682IFB1SSKL type=write size=1626661 2023-12-03T11:48:47.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.885+0000 7f24574f5780 1 < completed delete operation key=MY3K9M4QRS3L tag=EXCH1VJVYCA2 type=del 2023-12-03T11:48:47.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.886+0000 7f24574f5780 1 > prepared operation key=S0EH0Q0UA8HG tag=CUHKGLWHBKMS type=del size=2118765 2023-12-03T11:48:47.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.887+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~FPAW4CUEM18S.0 size=1048576 2023-12-03T11:48:47.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.888+0000 7f24574f5780 1 < completed part key=_multipart_DA6ZXCLKTC1L.2~FPAW4CUEM18S.1 size=673471 2023-12-03T11:48:47.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.890+0000 7f24574f5780 1 < completed multipart upload key=DA6ZXCLKTC1L upload=FPAW4CUEM18S tag=TW931KCEVSW5 type=write size=2770623 2023-12-03T11:48:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:48:47.891+0000 7f24574f5780 1 > prepared operation key=MY3K9M4QRS3L tag=UJEK2KAW30DE type=write size=771858 2023-12-03T11:48:47.893 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] cls_rgw_stats.simulate (8867 ms) 2023-12-03T11:48:47.893 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 1 test from cls_rgw_stats (8867 ms total) 2023-12-03T11:48:47.893 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:48:47.893 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:48:48.820 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 1 test from 1 test suite ran. (11667 ms total) 2023-12-03T11:48:48.820 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 1 test. 2023-12-03T11:48:48.824 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:48:48.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:48:48.895 INFO:tasks.workunit:Running workunits matching cls/test_cls_cmpomap.sh on client.0... 2023-12-03T11:48:48.896 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2023-12-03T11:48:48.896 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2023-12-03T11:48:48.968 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:48:48.968 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 31 tests from 1 test suite. 2023-12-03T11:48:48.968 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:48:50.827 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 31 tests from CmpOmap 2023-12-03T11:48:50.827 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2023-12-03T11:48:50.841 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (14 ms) 2023-12-03T11:48:50.841 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2023-12-03T11:48:50.849 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (9 ms) 2023-12-03T11:48:50.850 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2023-12-03T11:48:50.853 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (3 ms) 2023-12-03T11:48:50.853 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2023-12-03T11:48:50.856 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (3 ms) 2023-12-03T11:48:50.856 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_str 2023-12-03T11:48:50.862 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_str (6 ms) 2023-12-03T11:48:50.862 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2023-12-03T11:48:50.867 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_u64 (6 ms) 2023-12-03T11:48:50.867 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2023-12-03T11:48:50.870 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (3 ms) 2023-12-03T11:48:50.870 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2023-12-03T11:48:50.874 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (4 ms) 2023-12-03T11:48:50.875 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_u64_empty_default 2023-12-03T11:48:50.878 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_u64_empty_default (4 ms) 2023-12-03T11:48:50.878 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2023-12-03T11:48:50.883 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (4 ms) 2023-12-03T11:48:50.883 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2023-12-03T11:48:50.883 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (1 ms) 2023-12-03T11:48:50.884 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2023-12-03T11:48:50.884 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (1 ms) 2023-12-03T11:48:50.884 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2023-12-03T11:48:50.892 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (7 ms) 2023-12-03T11:48:50.892 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2023-12-03T11:48:50.898 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (7 ms) 2023-12-03T11:48:50.898 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2023-12-03T11:48:50.904 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (6 ms) 2023-12-03T11:48:50.904 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2023-12-03T11:48:50.910 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (6 ms) 2023-12-03T11:48:50.910 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2023-12-03T11:48:50.918 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_str (8 ms) 2023-12-03T11:48:50.918 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2023-12-03T11:48:50.927 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (8 ms) 2023-12-03T11:48:50.927 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2023-12-03T11:48:50.930 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (3 ms) 2023-12-03T11:48:50.930 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2023-12-03T11:48:50.932 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (3 ms) 2023-12-03T11:48:50.932 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2023-12-03T11:48:50.933 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (1 ms) 2023-12-03T11:48:50.933 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2023-12-03T11:48:50.933 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (0 ms) 2023-12-03T11:48:50.933 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2023-12-03T11:48:50.940 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (7 ms) 2023-12-03T11:48:50.940 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2023-12-03T11:48:50.945 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (5 ms) 2023-12-03T11:48:50.945 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2023-12-03T11:48:50.952 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (7 ms) 2023-12-03T11:48:50.952 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2023-12-03T11:48:50.959 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (7 ms) 2023-12-03T11:48:50.959 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2023-12-03T11:48:50.962 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (2 ms) 2023-12-03T11:48:50.962 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2023-12-03T11:48:50.964 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (3 ms) 2023-12-03T11:48:50.964 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2023-12-03T11:48:50.965 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (1 ms) 2023-12-03T11:48:50.965 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2023-12-03T11:48:50.966 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (1 ms) 2023-12-03T11:48:50.966 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_empty 2023-12-03T11:48:50.973 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_empty (7 ms) 2023-12-03T11:48:50.973 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 31 tests from CmpOmap (147 ms total) 2023-12-03T11:48:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:48:50.973 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:48:51.838 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 31 tests from 1 test suite ran. (2870 ms total) 2023-12-03T11:48:51.838 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 31 tests. 2023-12-03T11:48:51.840 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:48:51.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:48:51.912 INFO:tasks.workunit:Running workunits matching cls/test_cls_2pc_queue.sh on client.0... 2023-12-03T11:48:51.913 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2023-12-03T11:48:51.913 DEBUG:teuthology.orchestra.run.smithi116:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-12-03T11:48:51.986 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:48:51.986 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:48:51.986 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:48:51.986 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from TestCls2PCQueue 2023-12-03T11:48:51.986 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2023-12-03T11:48:54.858 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2871 ms) 2023-12-03T11:48:54.859 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2023-12-03T11:48:57.861 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3002 ms) 2023-12-03T11:48:57.862 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.Reserve 2023-12-03T11:49:00.888 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.Reserve (3027 ms) 2023-12-03T11:49:00.888 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2023-12-03T11:49:03.950 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3064 ms) 2023-12-03T11:49:03.950 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.Commit 2023-12-03T11:49:06.978 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.Commit (3027 ms) 2023-12-03T11:49:06.978 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.Stats 2023-12-03T11:49:09.994 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.Stats (3016 ms) 2023-12-03T11:49:09.994 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.UpgradeFromReef 2023-12-03T11:49:14.023 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.UpgradeFromReef (4030 ms) 2023-12-03T11:49:14.023 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.Abort 2023-12-03T11:49:17.071 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.Abort (3047 ms) 2023-12-03T11:49:17.071 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2023-12-03T11:49:20.085 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.ReserveError (3014 ms) 2023-12-03T11:49:20.085 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.CommitError 2023-12-03T11:49:23.111 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.CommitError (3026 ms) 2023-12-03T11:49:23.111 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.AbortError 2023-12-03T11:49:26.133 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.AbortError (3023 ms) 2023-12-03T11:49:26.134 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2023-12-03T11:49:29.150 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3016 ms) 2023-12-03T11:49:29.150 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2023-12-03T11:49:32.173 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3023 ms) 2023-12-03T11:49:32.173 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2023-12-03T11:49:35.195 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3023 ms) 2023-12-03T11:49:35.196 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2023-12-03T11:49:38.217 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3019 ms) 2023-12-03T11:49:38.217 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2023-12-03T11:49:41.245 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3031 ms) 2023-12-03T11:49:41.245 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2023-12-03T11:49:44.286 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3041 ms) 2023-12-03T11:49:44.286 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2023-12-03T11:49:47.303 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.Cleanup (3017 ms) 2023-12-03T11:49:47.303 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2023-12-03T11:49:57.837 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.MultiProducer (10533 ms) 2023-12-03T11:49:57.837 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2023-12-03T11:50:01.857 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (4020 ms) 2023-12-03T11:50:01.857 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2023-12-03T11:50:14.898 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (13041 ms) 2023-12-03T11:50:14.898 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2023-12-03T11:50:18.916 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (4018 ms) 2023-12-03T11:50:18.916 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2023-12-03T11:50:24.938 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (6022 ms) 2023-12-03T11:50:24.938 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2023-12-03T11:50:30.030 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (5092 ms) 2023-12-03T11:50:30.031 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from TestCls2PCQueue (98044 ms total) 2023-12-03T11:50:30.031 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:30.031 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:30.031 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 24 tests from 1 test suite ran. (98044 ms total) 2023-12-03T11:50:30.031 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 24 tests. 2023-12-03T11:50:30.034 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:50:30.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:50:30.107 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2023-12-03T11:50:30.109 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2023-12-03T11:50:30.110 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2023-12-03T11:50:30.186 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:30.186 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:30.187 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 6 tests from 1 test suite. 2023-12-03T11:50:30.187 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:32.045 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 6 tests from rgw_gc_log 2023-12-03T11:50:32.045 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2023-12-03T11:50:32.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.init_existing_queue (281 ms) 2023-12-03T11:50:32.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2023-12-03T11:50:32.329 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.init_existing_omap (4 ms) 2023-12-03T11:50:32.329 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2023-12-03T11:50:32.332 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (2 ms) 2023-12-03T11:50:32.332 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2023-12-03T11:50:32.334 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (2 ms) 2023-12-03T11:50:32.334 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2023-12-03T11:50:32.336 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.defer1_after_init (3 ms) 2023-12-03T11:50:32.336 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2023-12-03T11:50:32.338 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] rgw_gc_log.defer2_before_init (1 ms) 2023-12-03T11:50:32.338 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 6 tests from rgw_gc_log (293 ms total) 2023-12-03T11:50:32.338 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:32.338 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:33.059 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 6 tests from 1 test suite ran. (2872 ms total) 2023-12-03T11:50:33.059 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 6 tests. 2023-12-03T11:50:33.063 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:50:33.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:50:33.137 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_obj.sh on client.0... 2023-12-03T11:50:33.139 INFO:tasks.workunit:Running workunit rgw/test_rgw_obj.sh... 2023-12-03T11:50:33.139 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/test_rgw_obj.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_obj.sh 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 4 tests from 1 test suite. 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from TestRGWObj 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestRGWObj.underscore 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i= 2023-12-03T11:50:33.221 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i= 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i= 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.222 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.223 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.224 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.225 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.226 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.227 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i= 2023-12-03T11:50:33.228 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i= 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i= 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.229 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.230 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.231 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.232 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.233 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i=v1 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i=v1 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns= i=v1 2023-12-03T11:50:33.234 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.235 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.236 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.237 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.238 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.239 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.240 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: _obj[v1] 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.241 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.242 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.243 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.244 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.245 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.246 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_obj", 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestRGWObj.underscore (0 ms) 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestRGWObj.no_underscore 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i= 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i= 2023-12-03T11:50:33.247 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i= 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.248 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.249 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.250 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.251 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.252 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i= 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i= 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i= 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.253 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.254 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.255 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.256 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.257 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.258 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.259 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i=v1 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i=v1 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns= i=v1 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.260 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.261 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.262 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.263 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.264 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:parsed: obj[v1] 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout: "obj1": { 2023-12-03T11:50:33.265 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.266 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "obj2": { 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.267 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3": { 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.268 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "obj3-2": { 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.269 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "obj", 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "v1", 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.270 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestRGWObj.no_underscore (0 ms) 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestRGWObj.obj_to_raw 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.271 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.272 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker_myobj", 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.273 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.274 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.275 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.276 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.277 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker_myobj", 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.278 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.279 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.280 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:50:33.281 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.282 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.283 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.284 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.285 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.286 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.287 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.288 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.289 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.290 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.291 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.292 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.293 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.294 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.295 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.296 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.297 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.298 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.299 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.300 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.301 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.303 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.304 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.305 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj", 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.306 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.307 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.308 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.309 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj", 2023-12-03T11:50:33.310 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.311 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.312 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.313 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.314 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.315 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.316 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.317 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.318 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.319 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.320 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.322 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.324 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.326 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.327 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.328 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.329 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.331 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.333 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.335 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.336 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.338 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj_", 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.339 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.341 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.342 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj_", 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.343 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.344 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.345 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.346 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.347 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.348 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.349 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.350 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.351 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.352 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.353 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.354 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.355 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.356 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.357 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.358 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.359 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.360 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.361 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.362 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": "default-placement.data", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "test", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": "", 2023-12-03T11:50:33.363 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": "" 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.364 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "obj": { 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.365 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.366 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.367 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestRGWObj.obj_to_raw (0 ms) 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] TestRGWObj.old_to_raw 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker_myobj", 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.368 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.369 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker_myobj 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker_myobj", 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.370 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.371 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker_myobj 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.372 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.373 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns_myobj 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.374 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.375 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.377 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns_myobj 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.378 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__:inst_myobj 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.379 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.380 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__:inst_myobj 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:50:33.381 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.382 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.383 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns:inst_myobj 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.384 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.385 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.386 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "myobj", 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.387 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns:inst_myobj 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj", 2023-12-03T11:50:33.388 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.389 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.390 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker___myobj 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.391 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.392 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker___myobj 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:50:33.393 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.394 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns__myobj 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.395 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.396 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns__myobj 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.397 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.398 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__:inst__myobj 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj" 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.399 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.400 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__:inst__myobj 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.401 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.402 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns:inst__myobj 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.403 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns:inst__myobj 2023-12-03T11:50:33.404 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj_", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.405 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker___myobj_ 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.406 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker___myobj_", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.407 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker___myobj_ 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.408 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.409 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.410 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns__myobj_ 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.411 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.412 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "", 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.413 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns__myobj_ 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:50:33.414 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.415 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.416 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__:inst__myobj_ 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.417 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "marker__myobj_" 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.418 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "" 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.419 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__:inst__myobj_ 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj": { 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj": { 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.420 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.421 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout:raw=.data-pool:marker__ns:inst__myobj_ 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_obj2": { 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": ".data-pool", 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "loc": "" 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "new_obj2": { 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket": { 2023-12-03T11:50:33.422 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "ebtest", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "marker": "marker", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_id": "bucket-id", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "tenant": "", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "explicit_placement": { 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "data_pool": ".data-pool", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "data_extra_pool": "", 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "index_pool": ".index-pool" 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "key": { 2023-12-03T11:50:33.423 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "_myobj_", 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout: "instance": "inst", 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout: "ns": "ns" 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:raw2=.data-pool:marker__ns:inst__myobj_ 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] TestRGWObj.old_to_raw (0 ms) 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from TestRGWObj (0 ms total) 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 4 tests from 1 test suite ran. (0 ms total) 2023-12-03T11:50:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 4 tests. 2023-12-03T11:50:33.425 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:50:33.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:50:33.454 INFO:tasks.workunit:Running workunits matching rgw/test_librgw_file.sh on client.0... 2023-12-03T11:50:33.455 INFO:tasks.workunit:Running workunit rgw/test_librgw_file.sh... 2023-12-03T11:50:33.455 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/test_librgw_file.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh 2023-12-03T11:50:33.556 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:33.556 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:33.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.568+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 realm 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.570+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.571+0000 7f54128c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:50:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.571+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.572+0000 7f54128c0800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:50:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.572+0000 7f54128c0800 20 searching for the correct realm 2023-12-03T11:50:33.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f5407fff640 0 -- 172.21.15.116:0/4056444122 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x557cbddf7f80 msgr2=0x557cbddfa370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.583+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:50:33.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 zone default found 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 4 Realm: () 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 zonegroup default 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.584+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.585+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.585+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.585+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.585+0000 7f54128c0800 20 started sync module instance, tier type = 2023-12-03T11:50:33.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.585+0000 7f54128c0800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:50:33.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.592+0000 7f5410bc1640 0 -- 172.21.15.116:0/4056444122 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f53f801b860 msgr2=0x7f53f800c350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=0 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=1 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=2 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=3 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=4 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=5 2023-12-03T11:50:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=6 2023-12-03T11:50:33.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 20 add_watcher() i=7 2023-12-03T11:50:33.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.595+0000 7f54128c0800 2 all 8 watchers are set, enabling cache 2023-12-03T11:50:33.598 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.597+0000 7f53d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:50:33.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.597+0000 7f54128c0800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:50:33.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.597+0000 7f54128c0800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:50:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.603+0000 7f54128c0800 5 note: GC not initialized 2023-12-03T11:50:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.603+0000 7f53457fa640 20 reqs_thread_entry: start 2023-12-03T11:50:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.603+0000 7f54128c0800 20 init_complete bucket index max shards: 11 2023-12-03T11:50:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.603+0000 7f54128c0800 20 Filter name: none 2023-12-03T11:50:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f533effd640 20 reqs_thread_entry: start 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache get: name=default.rgw.meta+users.uid+ceph-test-librgw-file : miss 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x0 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 adding default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache get: name=default.rgw.meta+users.email+librgw@example.com : miss 2023-12-03T11:50:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 adding default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache get: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= : miss 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados->read ofs=0 len=0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= info.flags=0x0 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 adding default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= to cache LRU end 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache get: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= : hit (negative entry) 2023-12-03T11:50:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.604+0000 7f54128c0800 10 cache get: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= : hit (negative entry) 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f54128c0800 10 moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f54128c0800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:ceph-test-librgw-file, ofs0, ns] 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f5406ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1056561954817 cookie 93994249853168 notifier 5655 bl.length()=663 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f5406ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-12-03T11:50:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.606+0000 7f5406ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:50:33.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-12-03T11:50:33.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f54128c0800 10 moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:50:33.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f54128c0800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:librgw@example.com, ofs0, ns] 2023-12-03T11:50:33.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f5406ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1056561954819 cookie 93994249859840 notifier 5655 bl.length()=194 2023-12-03T11:50:33.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f5406ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-12-03T11:50:33.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.608+0000 7f5406ffd640 10 rgw watcher librados: moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54128c0800 10 cache put: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= info.flags=0x7 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54128c0800 10 moving default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= to cache LRU end 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54128c0800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:DQMgsQZAaScT41RcOT2RpQLJwmw=, ofs0, ns] 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54067fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1056561954816 cookie 93994249877312 notifier 5655 bl.length()=203 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54067fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= info.flags=0x7 2023-12-03T11:50:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.610+0000 7f54067fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+DQMgsQZAaScT41RcOT2RpQLJwmw= to cache LRU end 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "user_id": "ceph-test-librgw-file", 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "display_name": "librgw test user", 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "email": "librgw@example.com", 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "suspended": 0, 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "max_buckets": 1000, 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "subusers": [], 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "keys": [ 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-12-03T11:50:33.612 INFO:tasks.workunit.client.0.smithi116.stdout: "user": "ceph-test-librgw-file", 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "access_key": "DQMgsQZAaScT41RcOT2RpQLJwmw=", 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "secret_key": "v3iLe/IvP4fUkvayuM8+jRLKgEVwmNE0uZ7wYfLnC85GJvS65vnkxQ==" 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "swift_keys": [], 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "caps": [], 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "op_mask": "read, write, delete", 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "default_placement": "", 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "default_storage_class": "", 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "placement_tags": [], 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_quota": { 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "enabled": false, 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size": -1, 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: "max_objects": -1 2023-12-03T11:50:33.613 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "user_quota": { 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "enabled": false, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size": -1, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "max_objects": -1 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "temp_url_keys": [], 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "rgw", 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: "mfa_ids": [] 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:50:33.614 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:33.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.611+0000 7f53d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:50:33.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.613+0000 7f54128c0800 20 remove_watcher() i=0 2023-12-03T11:50:33.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.613+0000 7f54128c0800 2 removed watcher, disabling cache 2023-12-03T11:50:33.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.614+0000 7f54128c0800 20 remove_watcher() i=1 2023-12-03T11:50:33.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.615+0000 7f54128c0800 20 remove_watcher() i=2 2023-12-03T11:50:33.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.617+0000 7f54128c0800 20 remove_watcher() i=3 2023-12-03T11:50:33.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.618+0000 7f54128c0800 20 remove_watcher() i=4 2023-12-03T11:50:33.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.619+0000 7f54128c0800 20 remove_watcher() i=5 2023-12-03T11:50:33.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.620+0000 7f54128c0800 20 remove_watcher() i=6 2023-12-03T11:50:33.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:33.621+0000 7f54128c0800 20 remove_watcher() i=7 2023-12-03T11:50:33.626 INFO:tasks.workunit.client.0.smithi116.stdout:phase 1.1 2023-12-03T11:50:33.645 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:50:33.646 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:33.646 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:50:33.646 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:33.656 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:33.656 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:34.028 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (382 ms) 2023-12-03T11:50:34.028 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:50:34.029 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT_NOROOT (1 ms) 2023-12-03T11:50:34.029 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:34.029 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:50:34.029 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:50:34.040 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/quux 2023-12-03T11:50:34.045 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/f1 2023-12-03T11:50:34.048 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/f2 2023-12-03T11:50:34.051 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/f1 2023-12-03T11:50:34.054 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/d1/ 2023-12-03T11:50:34.057 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/hungry 2023-12-03T11:50:34.060 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/hungry/ 2023-12-03T11:50:34.064 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/momma 2023-12-03T11:50:34.067 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/bear/ 2023-12-03T11:50:34.070 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/sasquatch 2023-12-03T11:50:34.074 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/sasquatch/ 2023-12-03T11:50:34.077 INFO:tasks.workunit.client.0.smithi116.stdout:creating: nfsroot:foo/bar/baz/frobozz 2023-12-03T11:50:34.079 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_HIER1 (51 ms) 2023-12-03T11:50:34.079 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:50:34.188 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_DIRS1 (108 ms) 2023-12-03T11:50:34.188 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:50:34.188 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:50:34.188 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:50:34.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (20 ms) 2023-12-03T11:50:34.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:50:34.248 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (40 ms) 2023-12-03T11:50:34.248 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:50:34.248 INFO:tasks.workunit.client.0.smithi116.stdout:rename file rfile_0 to rfile_r0 (bucket brename0) 2023-12-03T11:50:34.254 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (6 ms) 2023-12-03T11:50:34.254 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:50:34.254 INFO:tasks.workunit.client.0.smithi116.stdout:rename file rfile_1 (bucket brename0) to rfile_rhilldog (bucket brename1) 2023-12-03T11:50:34.261 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (8 ms) 2023-12-03T11:50:34.261 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:50:34.261 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_0 2023-12-03T11:50:34.261 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_2 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:34.262 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:50:34.263 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:50:34.263 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_0 2023-12-03T11:50:34.263 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_0/sfile_0 2023-12-03T11:50:34.263 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:34.263 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_0/sfile_1 2023-12-03T11:50:34.265 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:34.265 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_1 2023-12-03T11:50:34.265 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_1/sfile_0 2023-12-03T11:50:34.266 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:34.266 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_1/sfile_1 2023-12-03T11:50:34.267 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:34.267 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_2 2023-12-03T11:50:34.268 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_2/sfile_0 2023-12-03T11:50:34.269 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:34.269 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_2/sfile_1 2023-12-03T11:50:34.270 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:34.270 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READ_DIRS1 (9 ms) 2023-12-03T11:50:34.270 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_SETUP1 (172 ms) 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:34.443 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-12-03T11:50:34.444 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:50:34.449 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/f2 (f2): FILE 2023-12-03T11:50:34.450 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/f1 (f1): FILE 2023-12-03T11:50:34.452 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-12-03T11:50:34.454 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-12-03T11:50:34.457 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:50:34.457 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:50:34.458 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-12-03T11:50:34.458 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-12-03T11:50:34.459 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:50:34.459 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:50:34.460 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar (bar): DIR 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo (foo): DIR 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot (nfsroot): DIR 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.HIER1 (19 ms) 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:50:34.462 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:34.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:34.460+0000 7f5f69f228c0 -1 shutting down 2023-12-03T11:50:34.464 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:34.461+0000 7f5f42f52640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97188, si_uid : 1000, si_addr0x3e800017ba4, si_status0 2023-12-03T11:50:34.583 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (122 ms) 2023-12-03T11:50:34.583 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW (938 ms total) 2023-12-03T11:50:34.583 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:34.584 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:34.584 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 24 tests from 1 test suite ran. (938 ms total) 2023-12-03T11:50:34.584 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 24 tests. 2023-12-03T11:50:34.589 INFO:tasks.workunit.client.0.smithi116.stdout:phase 1.2 2023-12-03T11:50:34.609 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:50:34.609 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:34.609 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:50:34.609 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:34.616 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:34.616 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:34.757 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (149 ms) 2023-12-03T11:50:34.757 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT_NOROOT (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_HIER1 (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_DIRS1 (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-12-03T11:50:34.759 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READ_DIRS1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:50:34.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.HIER1 (0 ms) 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:50:34.761 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (1 ms) 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:34.762 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:34.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:34.756+0000 7f39982c08c0 -1 shutting down 2023-12-03T11:50:34.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:34.757+0000 7f3970e38640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97298, si_uid : 1000, si_addr0x3e800017c12, si_status0 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (63 ms) 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW (213 ms total) 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 24 tests from 1 test suite ran. (213 ms total) 2023-12-03T11:50:34.822 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 24 tests. 2023-12-03T11:50:34.827 INFO:tasks.workunit.client.0.smithi116.stdout:phase 1.3 2023-12-03T11:50:34.846 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:50:34.846 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:34.846 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:50:34.846 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:34.855 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:34.855 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:35.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (365 ms) 2023-12-03T11:50:35.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:50:35.211 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT_NOROOT (1 ms) 2023-12-03T11:50:35.211 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:35.211 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:50:35.211 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:50:35.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_HIER1 (5 ms) 2023-12-03T11:50:35.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETUP_DIRS1 (59 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_0 2023-12-03T11:50:35.276 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout:scanning objects in /bdirs1/dir_2 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:50:35.277 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_0 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_0/sfile_0 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_0/sfile_1 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_1 2023-12-03T11:50:35.278 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_1/sfile_0 2023-12-03T11:50:35.279 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:35.279 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_1/sfile_1 2023-12-03T11:50:35.281 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:35.281 INFO:tasks.workunit.client.0.smithi116.stdout:read back objects in /bdirs1/dir_2 2023-12-03T11:50:35.281 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_2/sfile_0 2023-12-03T11:50:35.282 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:50:35.282 INFO:tasks.workunit.client.0.smithi116.stdout:reading 0,256 dir_2/sfile_1 2023-12-03T11:50:35.283 INFO:tasks.workunit.client.0.smithi116.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:50:35.283 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READ_DIRS1 (8 ms) 2023-12-03T11:50:35.283 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:50:35.284 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:50:35.285 INFO:tasks.workunit.client.0.smithi116.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:50:35.285 INFO:tasks.workunit.client.0.smithi116.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:50:35.285 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-12-03T11:50:35.285 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:50:35.290 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/f2 (f2): FILE 2023-12-03T11:50:35.290 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/f1 (f1): FILE 2023-12-03T11:50:35.293 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-12-03T11:50:35.294 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-12-03T11:50:35.297 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:50:35.297 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:50:35.298 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-12-03T11:50:35.298 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-12-03T11:50:35.299 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:50:35.299 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:50:35.299 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-12-03T11:50:35.301 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-12-03T11:50:35.301 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo/bar (bar): DIR 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot/foo (foo): DIR 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:/nfsroot (nfsroot): DIR 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.HIER1 (18 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:35.302 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:35.303 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:35.303 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:35.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:35.300+0000 7f24987238c0 -1 shutting down 2023-12-03T11:50:35.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:35.301+0000 7f2471775640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97408, si_uid : 1000, si_addr0x3e800017c80, si_status0 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (73 ms) 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 24 tests from LibRGW (529 ms total) 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 24 tests from 1 test suite ran. (529 ms total) 2023-12-03T11:50:35.375 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 24 tests. 2023-12-03T11:50:35.380 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2.1 2023-12-03T11:50:35.400 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:50:35.400 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:35.401 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from LibRGW 2023-12-03T11:50:35.401 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:35.751 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (351 ms) 2023-12-03T11:50:35.752 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:35.752 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:35.752 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:35.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-12-03T11:50:35.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:35.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:50:35.760 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-12-03T11:50:35.770 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave0 2023-12-03T11:50:35.782 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave1 2023-12-03T11:50:35.792 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave2 2023-12-03T11:50:35.801 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave3 2023-12-03T11:50:35.810 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave4 2023-12-03T11:50:35.817 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave5 2023-12-03T11:50:35.825 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave6 2023-12-03T11:50:35.834 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave7 2023-12-03T11:50:35.842 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave8 2023-12-03T11:50:35.851 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave9 2023-12-03T11:50:35.851 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (89 ms) 2023-12-03T11:50:35.851 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (258 ms) 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:36.108 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:36.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:36.106+0000 7fb4ffd428c0 -1 shutting down 2023-12-03T11:50:36.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:36.107+0000 7fb4d8e7f640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97518, si_uid : 1000, si_addr0x3e800017cee, si_status0 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (108 ms) 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from LibRGW (816 ms total) 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 9 tests from 1 test suite ran. (816 ms total) 2023-12-03T11:50:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 9 tests. 2023-12-03T11:50:36.221 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2.2 2023-12-03T11:50:36.241 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:50:36.241 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:36.242 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from LibRGW 2023-12-03T11:50:36.242 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:36.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (153 ms) 2023-12-03T11:50:36.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:36.395 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:36.395 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:36.395 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:50:36.395 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:36.422 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (27 ms) 2023-12-03T11:50:36.422 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-12-03T11:50:36.430 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave0 2023-12-03T11:50:36.439 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave1 2023-12-03T11:50:36.449 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave2 2023-12-03T11:50:36.458 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave3 2023-12-03T11:50:36.465 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave4 2023-12-03T11:50:36.473 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave5 2023-12-03T11:50:36.481 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave6 2023-12-03T11:50:36.488 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave7 2023-12-03T11:50:36.499 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave8 2023-12-03T11:50:36.509 INFO:tasks.workunit.client.0.smithi116.stdout:created: sorrydave9 2023-12-03T11:50:36.509 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (86 ms) 2023-12-03T11:50:36.509 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-12-03T11:50:36.764 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (256 ms) 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:36.763+0000 7ff8e81298c0 -1 shutting down 2023-12-03T11:50:36.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:36.764+0000 7ff8c11dc640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97629, si_uid : 1000, si_addr0x3e800017d5d, si_status0 2023-12-03T11:50:36.858 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (93 ms) 2023-12-03T11:50:36.858 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 9 tests from LibRGW (616 ms total) 2023-12-03T11:50:36.858 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:36.858 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:36.859 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 9 tests from 1 test suite ran. (616 ms total) 2023-12-03T11:50:36.859 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 9 tests. 2023-12-03T11:50:36.863 INFO:tasks.workunit.client.0.smithi116.stdout:phase 3.1 2023-12-03T11:50:37.179 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 13 tests from 1 test suite. 2023-12-03T11:50:37.179 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:37.179 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 13 tests from LibRGW 2023-12-03T11:50:37.179 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:37.188 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:37.188 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:37.341 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (162 ms) 2023-12-03T11:50:37.341 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:37.342 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:50:37.342 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:37.350 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-12-03T11:50:37.350 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:50:37.353 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-12-03T11:50:37.353 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:50:37.354 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2023-12-03T11:50:37.355 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LARGE1 2023-12-03T11:50:37.768 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LARGE1 (414 ms) 2023-12-03T11:50:37.768 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LARGE2 2023-12-03T11:50:37.820 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LARGE2 (52 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:37.819+0000 7fc30d7268c0 -1 shutting down 2023-12-03T11:50:37.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:37.821+0000 7fc2e6752640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97739, si_uid : 1000, si_addr0x3e800017dcb, si_status0 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (86 ms) 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 13 tests from LibRGW (727 ms total) 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 13 tests from 1 test suite ran. (727 ms total) 2023-12-03T11:50:37.906 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 13 tests. 2023-12-03T11:50:37.912 INFO:tasks.workunit.client.0.smithi116.stdout:phase 3.2 2023-12-03T11:50:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 13 tests from 1 test suite. 2023-12-03T11:50:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 13 tests from LibRGW 2023-12-03T11:50:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:38.235 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:38.235 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:38.393 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (165 ms) 2023-12-03T11:50:38.393 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:38.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:38.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:38.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:50:38.394 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:50:38.398 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2023-12-03T11:50:38.399 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:50:38.402 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (4 ms) 2023-12-03T11:50:38.402 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LARGE1 2023-12-03T11:50:38.462 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LARGE1 (60 ms) 2023-12-03T11:50:38.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LARGE2 2023-12-03T11:50:38.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LARGE2 (0 ms) 2023-12-03T11:50:38.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:50:38.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:50:38.463 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:50:38.466 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-12-03T11:50:38.466 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (25 ms) 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:38.491+0000 7fdacdd268c0 -1 shutting down 2023-12-03T11:50:38.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:38.491+0000 7fdaa6d24640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97849, si_uid : 1000, si_addr0x3e800017e39, si_status0 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (65 ms) 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 13 tests from LibRGW (329 ms total) 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 13 tests from 1 test suite ran. (329 ms total) 2023-12-03T11:50:38.557 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 13 tests. 2023-12-03T11:50:38.563 INFO:tasks.workunit.client.0.smithi116.stdout:phase 4.1 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from LibRGW 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.TVAR 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:read i1: 64001 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:read s1: blunderbuss 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-12-03T11:50:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:38.593 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:38.593 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:38.946 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (364 ms) 2023-12-03T11:50:38.947 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:38.947 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:38.947 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:50:38.963 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (16 ms) 2023-12-03T11:50:38.963 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_0 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_1 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_2 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_3 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_4 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_5 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_6 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_7 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_8 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_9 2023-12-03T11:50:39.538 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_10 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_11 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_12 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_13 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_14 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_15 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_16 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_17 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_18 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_19 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_20 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_21 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_22 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_23 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_24 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_25 2023-12-03T11:50:39.539 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_26 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_27 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_28 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_29 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_30 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_31 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_32 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_33 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_34 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_35 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_36 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_37 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_38 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_39 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_40 2023-12-03T11:50:39.540 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_41 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_42 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_43 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_44 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_45 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_46 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_47 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_48 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_49 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_50 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_51 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_52 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_53 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_54 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_55 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_56 2023-12-03T11:50:39.541 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_57 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_58 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_59 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_60 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_61 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_62 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_63 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_64 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_65 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_66 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_67 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_68 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_69 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_70 2023-12-03T11:50:39.542 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_71 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_72 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_73 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_74 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_75 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_76 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_77 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_78 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_79 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_80 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_81 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_82 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_83 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_84 2023-12-03T11:50:39.543 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_85 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_86 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_87 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_88 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_89 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_90 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_91 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_92 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_93 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_94 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_95 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_96 2023-12-03T11:50:39.544 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_97 2023-12-03T11:50:39.545 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_98 2023-12-03T11:50:39.545 INFO:tasks.workunit.client.0.smithi116.stdout:created: dmarker:f_99 2023-12-03T11:50:39.545 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (574 ms) 2023-12-03T11:50:39.545 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:50:39.546 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=. flags=4 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=.. flags=4 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:50:39.547 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:50:39.548 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:50:39.549 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:50:39.550 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:50:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:50:39.552 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_READDIR (3 ms) 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:50:39.553 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:50:39.554 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:50:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:50:39.556 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:50:39.557 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:50:39.558 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:50:39.559 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER2_READDIR (2 ms) 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:50:39.560 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:39.541+0000 7fc2a9ebf8c0 -1 shutting down 2023-12-03T11:50:39.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:39.542+0000 7fc284d9e640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97959, si_uid : 1000, si_addr0x3e800017ea7, si_status0 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (70 ms) 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from LibRGW (1030 ms total) 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 11 tests from 1 test suite ran. (1030 ms total) 2023-12-03T11:50:39.613 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 11 tests. 2023-12-03T11:50:39.619 INFO:tasks.workunit.client.0.smithi116.stdout:phase 4.2 2023-12-03T11:50:39.638 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:50:39.638 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:39.638 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from LibRGW 2023-12-03T11:50:39.638 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.TVAR 2023-12-03T11:50:39.638 INFO:tasks.workunit.client.0.smithi116.stdout:read i1: 64001 2023-12-03T11:50:39.639 INFO:tasks.workunit.client.0.smithi116.stdout:read s1: blunderbuss 2023-12-03T11:50:39.639 INFO:tasks.workunit.client.0.smithi116.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-12-03T11:50:39.639 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:40.001 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (363 ms) 2023-12-03T11:50:40.001 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:40.001 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:40.001 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:50:40.009 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (8 ms) 2023-12-03T11:50:40.010 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:50:40.010 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:50:40.010 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:50:40.010 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:50:40.010 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:50:40.012 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:50:40.013 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:50:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:50:40.015 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:50:40.016 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:50:40.017 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:50:40.018 INFO:tasks.workunit.client.0.smithi116.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER2_READDIR (3 ms) 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:40.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.012+0000 7fd77e8c58c0 -1 shutting down 2023-12-03T11:50:40.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.013+0000 7fd757415640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 98069, si_uid : 1000, si_addr0x3e800017f15, si_status0 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (52 ms) 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 11 tests from LibRGW (427 ms total) 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 11 tests from 1 test suite ran. (428 ms total) 2023-12-03T11:50:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 11 tests. 2023-12-03T11:50:40.070 INFO:tasks.workunit.client.0.smithi116.stdout:phase 5.1 2023-12-03T11:50:40.099 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 19 tests from 1 test suite. 2023-12-03T11:50:40.099 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:40.099 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 19 tests from LibRGW 2023-12-03T11:50:40.099 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:40.108 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:40.109 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:40.457 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (356 ms) 2023-12-03T11:50:40.457 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:40.457 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:40.457 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:40.464 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-12-03T11:50:40.464 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:50:40.467 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-12-03T11:50:40.467 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-12-03T11:50:40.468 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-12-03T11:50:40.468 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:50:40.469 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2023-12-03T11:50:40.469 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-12-03T11:50:40.469 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-12-03T11:50:40.469 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-12-03T11:50:40.473 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.PUT_OBJECT (4 ms) 2023-12-03T11:50:40.473 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-12-03T11:50:40.475 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.GET_OBJECT (2 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITEV 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READV 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READV (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:50:40.476 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.475+0000 7fe3426c28c0 -1 shutting down 2023-12-03T11:50:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.476+0000 7fe31b2db640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 98179, si_uid : 1000, si_addr0x3e800017f83, si_status0 2023-12-03T11:50:40.519 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (44 ms) 2023-12-03T11:50:40.520 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 19 tests from LibRGW (420 ms total) 2023-12-03T11:50:40.520 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:40.520 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:40.520 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 19 tests from 1 test suite ran. (420 ms total) 2023-12-03T11:50:40.520 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 19 tests. 2023-12-03T11:50:40.525 INFO:tasks.workunit.client.0.smithi116.stdout:phase 5.2 2023-12-03T11:50:40.553 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 19 tests from 1 test suite. 2023-12-03T11:50:40.553 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2023-12-03T11:50:40.553 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 19 tests from LibRGW 2023-12-03T11:50:40.553 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:50:40.560 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:50:40.560 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:50:40.716 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.INIT (163 ms) 2023-12-03T11:50:40.716 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:50:40.717 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:50:40.717 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:50:40.717 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:50:40.717 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:50:40.722 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2023-12-03T11:50:40.722 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-12-03T11:50:40.722 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.PUT_OBJECT (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.GET_OBJECT (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.WRITEV 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-12-03T11:50:40.723 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READV 2023-12-03T11:50:40.724 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READV (0 ms) 2023-12-03T11:50:40.724 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-12-03T11:50:40.724 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-12-03T11:50:40.724 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:50:40.726 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-12-03T11:50:40.727 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.DELETE_BUCKET (28 ms) 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:50:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:50:40.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.754+0000 7f514eda28c0 -1 shutting down 2023-12-03T11:50:40.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:50:40.755+0000 7f5129c83640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 98289, si_uid : 1000, si_addr0x3e800017ff1, si_status0 2023-12-03T11:50:40.781 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRGW.SHUTDOWN (25 ms) 2023-12-03T11:50:40.781 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 19 tests from LibRGW (226 ms total) 2023-12-03T11:50:40.781 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:50:40.781 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2023-12-03T11:50:40.781 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 19 tests from 1 test suite ran. (226 ms total) 2023-12-03T11:50:40.782 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 19 tests. 2023-12-03T11:50:40.786 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:50:40.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:50:40.814 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh'] on client.0... 2023-12-03T11:50:40.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:50:41.117 DEBUG:teuthology.parallel:result is None 2023-12-03T11:50:41.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:50:41.145 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:50:41.145 DEBUG:teuthology.orchestra.run.smithi116:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:50:41.200 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:50:41.200 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:50:41.211 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:50:41.211 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:50:41.211 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:50:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:50:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:50:41.256 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:50:41.311 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:50:41.311 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:50:41.366 INFO:tasks.workunit:timeout=3h 2023-12-03T11:50:41.366 INFO:tasks.workunit:cleanup=True 2023-12-03T11:50:41.366 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:50:41.421 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:51:47.202 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2023-12-03T11:51:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:51:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:51:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:51:47.205 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:51:47.260 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:51:47.262 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:51:47.262 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:51:47.312 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:51:47.361 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:51:47.400 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:51:47.401 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:51:47.402 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:51:47.440 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:51:47.443 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:51:47.443 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:51:47.498 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_s3_mp_reupload.sh on client.0... 2023-12-03T11:51:47.500 INFO:tasks.workunit:Running workunit rgw/test_rgw_s3_mp_reupload.sh... 2023-12-03T11:51:47.500 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/test_rgw_s3_mp_reupload.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_s3_mp_reupload.sh 2023-12-03T11:51:47.559 INFO:tasks.workunit.client.0.smithi116.stdout:INFO: fully qualified domain name: smithi116.front.sepia.ceph.com 2023-12-03T11:51:47.559 INFO:tasks.workunit.client.0.smithi116.stdout:== TESTING *ABORTING* MULTIPART UPLOAD WITH RE-UPLOADS == 2023-12-03T11:51:47.559 INFO:tasks.workunit.client.0.smithi116.stdout:== TESTING VERSIONED BUCKET == 2023-12-03T11:51:50.156 INFO:tasks.workunit.client.0.smithi116.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:51:50.355 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting pip 2023-12-03T11:51:50.361 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:51:50.408 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: pip 2023-12-03T11:51:50.409 INFO:tasks.workunit.client.0.smithi116.stdout: Attempting uninstall: pip 2023-12-03T11:51:50.409 INFO:tasks.workunit.client.0.smithi116.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:51:50.554 INFO:tasks.workunit.client.0.smithi116.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:51:50.559 INFO:tasks.workunit.client.0.smithi116.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:51:51.510 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed pip-23.3.1 2023-12-03T11:51:52.341 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting boto3 2023-12-03T11:51:52.342 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:51:52.849 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:51:52.850 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:51:52.910 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:51:52.911 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:51:52.964 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:51:52.966 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:51:53.032 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:51:53.033 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:51:53.115 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:51:53.117 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:51:53.321 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:51:53.322 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:51:53.345 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:51:53.346 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:51:53.366 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:51:53.367 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:51:53.466 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:51:54.546 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:51:54.761 INFO:tasks.workunit.client.0.smithi116.stdout:INFO: creating user 2023-12-03T11:51:54.857 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "user_id": "testid", 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "display_name": "M. Tester", 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "email": "tester@ceph.com", 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "suspended": 0, 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "max_buckets": 1000, 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "subusers": [], 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "keys": [ 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "user": "testid", 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "access_key": "0555b35654ad1656d804", 2023-12-03T11:51:54.858 INFO:tasks.workunit.client.0.smithi116.stdout: "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "swift_keys": [], 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "caps": [], 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "op_mask": "read, write, delete", 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "default_placement": "", 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "default_storage_class": "", 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "placement_tags": [], 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "bucket_quota": { 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "enabled": false, 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size": -1, 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "max_objects": -1 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:51:54.859 INFO:tasks.workunit.client.0.smithi116.stdout: "user_quota": { 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "enabled": false, 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size": -1, 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "max_objects": -1 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "temp_url_keys": [], 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "rgw", 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: "mfa_ids": [] 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-12-03T11:51:54.860 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-12-03T11:52:00.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:00.182 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:00.196 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:00.218 INFO:tasks.workunit.client.0.smithi116.stderr:5+0 records in 2023-12-03T11:52:00.218 INFO:tasks.workunit.client.0.smithi116.stderr:5+0 records out 2023-12-03T11:52:00.218 INFO:tasks.workunit.client.0.smithi116.stderr:5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0152679 s, 343 MB/s 2023-12-03T11:52:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:00.222 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:01.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:01.980 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.043 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.102 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.162 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.229 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.290 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.350 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.411 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.467 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.528 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.589 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.649 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.706 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi116.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:52:02.763 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:bucket nams is reupload-bkt-116220 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:endpoint is https://smithi116.front.sepia.ceph.com:443 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:start UploadId=2~xsW_CD7tc1GMOyRC3F_QYb9hJOO8Ptc 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 1 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 1 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 2} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.773 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.774 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.774 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.774 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.774 INFO:tasks.workunit.client.0.smithi116.stdout:phase 2 uploaded part {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1} 2023-12-03T11:52:02.774 INFO:tasks.workunit.client.0.smithi116.stdout:aborting multipart upload, parts=[{'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 1}, {'ETag': '"28e52450dddaeb3ad56d96bb0e86b11a"', 'PartNumber': 2}] 2023-12-03T11:52:03.283 INFO:tasks.workunit.client.0.smithi116.stdout:radosgw-admin bucket radoslist: 2023-12-03T11:52:03.544 INFO:tasks.workunit.client.0.smithi116.stdout:rados ls: 2023-12-03T11:52:03.620 INFO:tasks.workunit.client.0.smithi116.stdout:== PASSED == 2023-12-03T11:52:03.621 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:52:03.621 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:52:03.688 INFO:tasks.workunit:Stopping ['rgw/test_rgw_s3_mp_reupload.sh'] on client.0... 2023-12-03T11:52:03.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:52:04.019 DEBUG:teuthology.parallel:result is None 2023-12-03T11:52:04.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:52:04.047 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:52:04.047 DEBUG:teuthology.orchestra.run.smithi116:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:52:04.102 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:52:04.102 INFO:teuthology.run_tasks:Running task ragweed... 2023-12-03T11:52:04.115 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}} 2023-12-03T11:52:04.115 INFO:tasks.ragweed:Downloading ragweed... 2023-12-03T11:52:04.116 INFO:tasks.ragweed:Using branch 'wip-alimasa-multi-pubsub-etag' for ragweed 2023-12-03T11:52:04.116 DEBUG:teuthology.orchestra.run.smithi116:> git clone -b wip-alimasa-multi-pubsub-etag https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:52:04.157 INFO:teuthology.orchestra.run.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2023-12-03T11:52:04.202 INFO:teuthology.orchestra.run.smithi116.stderr:fatal: Remote branch wip-alimasa-multi-pubsub-etag not found in upstream origin 2023-12-03T11:52:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-12-03T11:52:04.204 INFO:tasks.ragweed:Using branch 'ceph-master' for ragweed 2023-12-03T11:52:04.204 DEBUG:teuthology.orchestra.run.smithi116:> git clone -b ceph-master https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:52:04.259 INFO:teuthology.orchestra.run.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2023-12-03T11:52:04.380 INFO:tasks.ragweed:Creating rgw users... 2023-12-03T11:52:04.380 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2023-12-03T11:52:04.381 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key UTBOVHAHAGVNNTWYJBFD --secret U0Bp8jBcWY+ZQi6Q6tqdL9EmGCu+WwK53RKhqq9krz3Wlz/+kb/+rg== --email ragweed.client.0+test@test.test --system false 2023-12-03T11:52:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:52:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:52:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.480+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 realm 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.482+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.483+0000 7fb1a212d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.483+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.484+0000 7fb1a212d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:52:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.484+0000 7fb1a212d800 20 searching for the correct realm 2023-12-03T11:52:04.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.494+0000 7fb19dd7e640 0 -- 172.21.15.116:0/2239175417 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5583cbacfde0 msgr2=0x5583cbad2230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.495+0000 7fb1a212d800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.495+0000 7fb1a212d800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.495+0000 7fb1a212d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.495+0000 7fb1a212d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.495+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.496+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.496+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.496+0000 7fb1a212d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.496+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.496+0000 7fb1a212d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 zone default found 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 4 Realm: () 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 zonegroup default 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 started sync module instance, tier type = 2023-12-03T11:52:04.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.497+0000 7fb1a212d800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:52:04.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.503+0000 7fb19e57f640 0 -- 172.21.15.116:0/2239175417 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fb194080380 msgr2=0x7fb194080860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=0 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=1 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=2 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=3 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=4 2023-12-03T11:52:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=5 2023-12-03T11:52:04.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.506+0000 7fb1a212d800 20 add_watcher() i=6 2023-12-03T11:52:04.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.507+0000 7fb1a212d800 20 add_watcher() i=7 2023-12-03T11:52:04.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.507+0000 7fb1a212d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:52:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.509+0000 7fb1637e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.509+0000 7fb1a212d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:52:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.509+0000 7fb1a212d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:52:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 5 note: GC not initialized 2023-12-03T11:52:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb0d27fc640 20 reqs_thread_entry: start 2023-12-03T11:52:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 20 init_complete bucket index max shards: 11 2023-12-03T11:52:04.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 20 Filter name: none 2023-12-03T11:52:04.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb0c3fff640 20 reqs_thread_entry: start 2023-12-03T11:52:04.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2023-12-03T11:52:04.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.514+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 cache get: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD : miss 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD info.flags=0x0 2023-12-03T11:52:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 adding default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD to cache LRU end 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 cache get: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD : hit (negative entry) 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.515+0000 7fb1a212d800 10 cache get: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD : hit (negative entry) 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.516+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.516+0000 7fb1a212d800 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.516+0000 7fb1a212d800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.517+0000 7fb17ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856725 cookie 94024546229840 notifier 5726 bl.length()=641 2023-12-03T11:52:04.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.517+0000 7fb17ffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-12-03T11:52:04.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.517+0000 7fb17ffff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:52:04.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.518+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-12-03T11:52:04.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.518+0000 7fb1a212d800 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.518+0000 7fb1a212d800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2023-12-03T11:52:04.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.519+0000 7fb17f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856710 cookie 94024546224112 notifier 5726 bl.length()=202 2023-12-03T11:52:04.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.519+0000 7fb17f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-12-03T11:52:04.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.519+0000 7fb17f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb1a212d800 10 cache put: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD info.flags=0x7 2023-12-03T11:52:04.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb1a212d800 10 moving default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD to cache LRU end 2023-12-03T11:52:04.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb1a212d800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:UTBOVHAHAGVNNTWYJBFD, ofs0, ns] 2023-12-03T11:52:04.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb17ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856711 cookie 94024546224112 notifier 5726 bl.length()=190 2023-12-03T11:52:04.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb17ffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD info.flags=0x7 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.521+0000 7fb17ffff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+UTBOVHAHAGVNNTWYJBFD to cache LRU end 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "user_id": "ragweed.client.0", 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "display_name": "Mr. ragweed.client.0", 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "email": "ragweed.client.0+test@test.test", 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "suspended": 0, 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max_buckets": 1000, 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "subusers": [], 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: "keys": [ 2023-12-03T11:52:04.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "user": "ragweed.client.0", 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "UTBOVHAHAGVNNTWYJBFD", 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "U0Bp8jBcWY+ZQi6Q6tqdL9EmGCu+WwK53RKhqq9krz3Wlz/+kb/+rg==" 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "swift_keys": [], 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": [], 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "op_mask": "read, write, delete", 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_storage_class": "", 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_tags": [], 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:52:04.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "temp_url_keys": [], 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "rgw", 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: "mfa_ids": [] 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-12-03T11:52:04.525 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:52:04.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.522+0000 7fb1637e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:04.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.524+0000 7fb1a212d800 20 remove_watcher() i=0 2023-12-03T11:52:04.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.524+0000 7fb1a212d800 2 removed watcher, disabling cache 2023-12-03T11:52:04.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.526+0000 7fb1a212d800 20 remove_watcher() i=1 2023-12-03T11:52:04.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.527+0000 7fb1a212d800 20 remove_watcher() i=2 2023-12-03T11:52:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.528+0000 7fb1a212d800 20 remove_watcher() i=3 2023-12-03T11:52:04.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.529+0000 7fb1a212d800 20 remove_watcher() i=4 2023-12-03T11:52:04.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.530+0000 7fb1a212d800 20 remove_watcher() i=5 2023-12-03T11:52:04.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.531+0000 7fb1a212d800 20 remove_watcher() i=6 2023-12-03T11:52:04.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.532+0000 7fb1a212d800 20 remove_watcher() i=7 2023-12-03T11:52:04.538 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2023-12-03T11:52:04.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key ABIGYRFREBRMBKJXURTF --secret 7t8w4HWKPMcQK066tfwGNOz63sTWeX+2s23O3Yipa+AqcZDqJNj2MQ== --email sysuser.client.0+test@test.test --system true 2023-12-03T11:52:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:52:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:52:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.598+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 realm 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.600+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.601+0000 7f11af922800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.601+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.601+0000 7f11af922800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:52:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.601+0000 7f11af922800 20 searching for the correct realm 2023-12-03T11:52:04.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.611+0000 7f11aada5640 0 -- 172.21.15.116:0/1000783575 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55f529c51230 msgr2=0x55f529c53620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:52:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:52:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:52:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:52:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.612+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 zone default found 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 4 Realm: () 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:52:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 zonegroup default 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 started sync module instance, tier type = 2023-12-03T11:52:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.613+0000 7f11af922800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:52:04.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.620+0000 7f11abda7640 0 -- 172.21.15.116:0/1000783575 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f11a0057eb0 msgr2=0x7f11a006b8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=0 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=1 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=2 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=3 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=4 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=5 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.622+0000 7f11af922800 20 add_watcher() i=6 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.623+0000 7f11af922800 20 add_watcher() i=7 2023-12-03T11:52:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.623+0000 7f11af922800 2 all 8 watchers are set, enabling cache 2023-12-03T11:52:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.624+0000 7f11717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.625+0000 7f11af922800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:52:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.625+0000 7f11af922800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:52:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.629+0000 7f11af922800 5 note: GC not initialized 2023-12-03T11:52:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.629+0000 7f10dffff640 20 reqs_thread_entry: start 2023-12-03T11:52:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.629+0000 7f11af922800 20 init_complete bucket index max shards: 11 2023-12-03T11:52:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.629+0000 7f11af922800 20 Filter name: none 2023-12-03T11:52:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f10ddffb640 20 reqs_thread_entry: start 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2023-12-03T11:52:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 10 cache get: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF : miss 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.630+0000 7f11af922800 20 rados->read ofs=0 len=0 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.631+0000 7f11af922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:04.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.631+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF info.flags=0x0 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.631+0000 7f11af922800 10 adding default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF to cache LRU end 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.631+0000 7f11af922800 10 cache get: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF : hit (negative entry) 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.631+0000 7f11af922800 10 cache get: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF : hit (negative entry) 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f11af922800 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f11af922800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f1191ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856712 cookie 94511456146688 notifier 6054 bl.length()=641 2023-12-03T11:52:04.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f1191ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.632+0000 7f1191ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f11af922800 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f11af922800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f1191ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856726 cookie 94511456136960 notifier 6054 bl.length()=202 2023-12-03T11:52:04.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f1191ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-12-03T11:52:04.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.634+0000 7f1191ffb640 10 rgw watcher librados: moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:52:04.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11af922800 10 cache put: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF info.flags=0x7 2023-12-03T11:52:04.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11af922800 10 moving default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF to cache LRU end 2023-12-03T11:52:04.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11af922800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:ABIGYRFREBRMBKJXURTF, ofs0, ns] 2023-12-03T11:52:04.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11917fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1069446856733 cookie 94511456155840 notifier 6054 bl.length()=190 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11917fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF info.flags=0x7 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.636+0000 7f11917fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+ABIGYRFREBRMBKJXURTF to cache LRU end 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "user_id": "sysuser.client.0", 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "display_name": "Mr. sysuser.client.0", 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "email": "sysuser.client.0+test@test.test", 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "suspended": 0, 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max_buckets": 1000, 2023-12-03T11:52:04.638 INFO:teuthology.orchestra.run.smithi116.stdout: "subusers": [], 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "keys": [ 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "user": "sysuser.client.0", 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "ABIGYRFREBRMBKJXURTF", 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "7t8w4HWKPMcQK066tfwGNOz63sTWeX+2s23O3Yipa+AqcZDqJNj2MQ==" 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "swift_keys": [], 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": [], 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "op_mask": "read, write, delete", 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "system": true, 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_storage_class": "", 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_tags": [], 2023-12-03T11:52:04.639 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "temp_url_keys": [], 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "rgw", 2023-12-03T11:52:04.640 INFO:teuthology.orchestra.run.smithi116.stdout: "mfa_ids": [] 2023-12-03T11:52:04.641 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-12-03T11:52:04.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:52:04.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.637+0000 7f11717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:04.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.639+0000 7f11af922800 20 remove_watcher() i=0 2023-12-03T11:52:04.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.639+0000 7f11af922800 2 removed watcher, disabling cache 2023-12-03T11:52:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.640+0000 7f11af922800 20 remove_watcher() i=1 2023-12-03T11:52:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.641+0000 7f11af922800 20 remove_watcher() i=2 2023-12-03T11:52:04.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.642+0000 7f11af922800 20 remove_watcher() i=3 2023-12-03T11:52:04.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.643+0000 7f11af922800 20 remove_watcher() i=4 2023-12-03T11:52:04.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.644+0000 7f11af922800 20 remove_watcher() i=5 2023-12-03T11:52:04.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.645+0000 7f11af922800 20 remove_watcher() i=6 2023-12-03T11:52:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:04.646+0000 7f11af922800 20 remove_watcher() i=7 2023-12-03T11:52:04.652 INFO:tasks.ragweed:Configuring ragweed... 2023-12-03T11:52:04.653 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:52:04.653 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/archive/ragweed.client.0.conf 2023-12-03T11:52:04.667 INFO:tasks.ragweed:Configuring boto... 2023-12-03T11:52:04.667 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:52:04.667 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-12-03T11:52:04.722 DEBUG:teuthology.orchestra.run.smithi116:ragweed tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/ragweed.client.0 && RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox --sitepackages -- -v -m 'not fails_on_rgw' 2023-12-03T11:52:04.924 INFO:teuthology.orchestra.run.smithi116.stdout:GLOB sdist-make: /home/ubuntu/cephtest/ragweed.client.0/setup.py 2023-12-03T11:52:05.091 INFO:teuthology.orchestra.run.smithi116.stdout:py create: /home/ubuntu/cephtest/ragweed.client.0/.tox/py 2023-12-03T11:52:05.588 INFO:teuthology.orchestra.run.smithi116.stdout:py installdeps: -rrequirements.txt 2023-12-03T11:52:13.889 INFO:teuthology.orchestra.run.smithi116.stdout:py inst: /home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip 2023-12-03T11:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout:py installed: asyncssh==2.13.2,Babel==2.9.1,bcrypt==3.2.2,boto==2.49.0,cachetools==4.2.4,ceph==1.0.0,ceph-volume==1.0.0,cephfs==2.0.0,certifi==2021.10.8,cffi==1.14.5,chardet==4.0.0,cheroot==8.6.0,CherryPy==18.6.1,cockpit @ file:///builddir/build/BUILD/cockpit-299/tmp/wheel/cockpit-299-py3-none-any.whl,cryptography==36.0.1,dasbus==1.4,dbus-python==1.2.18,distro==1.5.0,docopt==0.6.2,exceptiongroup==1.2.0,execnet==2.0.2,file-magic==0.4.0,gevent==23.9.1,google-auth==2.23.4,gpg==1.15.1,greenlet==3.0.1,httplib2==0.22.0,idna==2.10,iniconfig==2.0.0,isodate==0.6.1,jaraco.classes==3.2.1,jaraco.collections==3.0.0,jaraco.functools==3.5.0,jaraco.text==3.2.0,Jinja2==2.11.3,jsonpatch==1.21,jsonpointer==2.0,kubernetes==26.1.0,libcomps==0.1.18,logutils==0.3.5,lxml==4.6.5,Mako==1.1.4.dev0,MarkupSafe==1.1.1,more-itertools==8.12.0,munch==4.0.0,natsort==7.1.1,nftables==0.1,numpy==1.20.1,oauthlib==3.1.1,packaging==23.2,pecan==1.4.2,perf==0.1,pexpect==4.8.0,pluggy==1.3.0,ply==3.11,portend==3.1.0,prettytable==0.7.2,psutil==5.8.0,ptyprocess==0.6.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.20,PyGObject==3.40.1,PyJWT==2.4.0,pyOpenSSL==21.0.0,pyparsing==2.4.7,PySocks==1.7.1,pytest==7.4.3,python-dateutil==2.8.1,python-linux-procfs==0.7.0,pytz==2021.1,pyudev==0.22.0,PyYAML==5.4.1,rados==2.0.0,ragweed @ file:///home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip#sha256=fb0afac6711287f910c5ce7f0eb4adb9ca465d2ad8c87d4be0b13b97e629c2af,rbd==2.0.0,registries-conf-ctl==1.0,remoto==1.2.1,repoze.lru==0.7,requests==2.25.1,requests-oauthlib==1.3.0,rgw==2.0.0,Routes==2.5.1,rpm==4.16.1.3,rsa==4.9,s3cmd==2.3.0,scipy==1.6.2,selinux==3.5,sepolicy==3.5,setools==4.4.3,setroubleshoot @ file:///builddir/build/BUILD/setroubleshoot-3.3.32/src,six==1.15.0,sos==4.3,systemd-python==234,tempora==5.0.0,toml==0.10.2,tomli==2.0.1,typing_extensions==4.4.0,urllib3==1.26.5,WebOb==1.8.7,websocket-client==1.2.3,Werkzeug==2.0.3,zc.lockfile==2.0,zope.event==5.0,zope.interface==6.1 2023-12-03T11:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout:py run-test-pre: PYTHONHASHSEED='330503497' 2023-12-03T11:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout:py run-test: commands[0] | python -m pytest -v -m 'not fails_on_rgw' 2023-12-03T11:52:16.134 INFO:teuthology.orchestra.run.smithi116.stdout:============================= test session starts ============================== 2023-12-03T11:52:16.134 INFO:teuthology.orchestra.run.smithi116.stdout:platform linux -- Python 3.9.17, pytest-7.4.3, pluggy-1.3.0 -- /home/ubuntu/cephtest/ragweed.client.0/.tox/py/bin/python 2023-12-03T11:52:16.134 INFO:teuthology.orchestra.run.smithi116.stdout:cachedir: .tox/py/.pytest_cache 2023-12-03T11:52:16.134 INFO:teuthology.orchestra.run.smithi116.stdout:rootdir: /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:52:16.134 INFO:teuthology.orchestra.run.smithi116.stdout:configfile: pytest.ini 2023-12-03T11:52:16.233 INFO:teuthology.orchestra.run.smithi116.stdout:collecting ... collected 9 items 2023-12-03T11:52:16.233 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:52:18.599 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_small_obj_data::test PASSED [ 11%] 2023-12-03T11:52:19.483 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_multipart_simple::test PASSED [ 22%] 2023-12-03T11:52:20.329 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_multipart_index_versioning::test PASSED [ 33%] 2023-12-03T11:52:21.144 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_multipart_defer_complete::test PASSED [ 44%] 2023-12-03T11:52:21.969 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_multipart_defer_update_complete::test PASSED [ 55%] 2023-12-03T11:52:28.775 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_obj_storage_class::test PASSED [ 66%] 2023-12-03T11:52:30.478 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart::test PASSED [ 77%] 2023-12-03T11:52:32.717 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_copy::test PASSED [ 88%] 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart_copy::test PASSED [100%] 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout:=============================== warnings summary =============================== 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout:.tox/py/lib/python3.9/site-packages/boto/plugin.py:40 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: /home/ubuntu/cephtest/ragweed.client.0/.tox/py/lib/python3.9/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses 2023-12-03T11:52:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: import imp 2023-12-03T11:52:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:52:36.031 INFO:teuthology.orchestra.run.smithi116.stdout:-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2023-12-03T11:52:36.031 INFO:teuthology.orchestra.run.smithi116.stdout:======================== 9 passed, 1 warning in 19.90s ========================= 2023-12-03T11:52:36.063 INFO:teuthology.orchestra.run.smithi116.stdout:___________________________________ summary ____________________________________ 2023-12-03T11:52:36.063 INFO:teuthology.orchestra.run.smithi116.stdout: py: commands succeeded 2023-12-03T11:52:36.063 INFO:teuthology.orchestra.run.smithi116.stdout: congratulations :) 2023-12-03T11:52:36.077 INFO:tasks.ragweed:Cleaning up boto... 2023-12-03T11:52:36.078 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/boto.cfg 2023-12-03T11:52:36.092 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid ragweed.client.0 --purge-data 2023-12-03T11:52:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:52:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:52:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.182+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 realm 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.184+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.185+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.185+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.186+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:52:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.186+0000 7f1d29525800 20 searching for the correct realm 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d25955640 0 -- 172.21.15.116:0/2005988185 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f1d1c056a70 msgr2=0x7f1d1c055be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.197+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 zone default found 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 4 Realm: () 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 zonegroup default 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.198+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.199+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.199+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.199+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.199+0000 7f1d29525800 20 started sync module instance, tier type = 2023-12-03T11:52:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.199+0000 7f1d29525800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=0 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=1 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=2 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=3 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=4 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=5 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d24953640 0 -- 172.21.15.116:0/2005988185 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x555f4e5b8750 msgr2=0x555f4e5bab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.208+0000 7f1d29525800 20 add_watcher() i=6 2023-12-03T11:52:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.209+0000 7f1d29525800 20 add_watcher() i=7 2023-12-03T11:52:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.209+0000 7f1d29525800 2 all 8 watchers are set, enabling cache 2023-12-03T11:52:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.211+0000 7f1ceafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.211+0000 7f1d29525800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:52:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.211+0000 7f1d29525800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:52:36.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1c59ffb640 20 reqs_thread_entry: start 2023-12-03T11:52:36.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1d29525800 20 init_complete bucket index max shards: 11 2023-12-03T11:52:36.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1d29525800 20 Filter name: none 2023-12-03T11:52:36.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1c4b7fe640 20 reqs_thread_entry: start 2023-12-03T11:52:36.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1d29525800 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.253+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.254+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=452 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.254+0000 7f1d29525800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x13 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.254+0000 7f1d29525800 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.254+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 : miss 2023-12-03T11:52:36.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 info.flags=0x17 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 to cache LRU end 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.255+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 2023-12-03T11:52:36.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.256+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart_copy 2023-12-03T11:52:36.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.256+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_multipart_index_versioning 2023-12-03T11:52:36.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.256+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_copy 2023-12-03T11:52:36.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.257+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_update_complete 2023-12-03T11:52:36.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.257+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_complete 2023-12-03T11:52:36.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.257+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class 2023-12-03T11:52:36.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.258+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_multipart_simple 2023-12-03T11:52:36.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.258+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart 2023-12-03T11:52:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.259+0000 7f1d29525800 10 cls_bucket_list_unordered: got tests/r_test_small_obj_data 2023-12-03T11:52:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.259+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x555f4e6ab498 s->prefetch_data=0 2023-12-03T11:52:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.260+0000 7f1d29525800 10 manifest: total_size = 5004 2023-12-03T11:52:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.260+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1498367493871780594 2023-12-03T11:52:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.260+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x555f4e6ab498 s->prefetch_data=0 2023-12-03T11:52:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.260+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6ab498 2023-12-03T11:52:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.260+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.1 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5004 stripe_ofs=5004 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1498367493871780594 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.g2NzUYW99OQK7sGVFc2NlUROzKM7Kro_0 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.263+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1498367493871780594 2023-12-03T11:52:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart_copy 2023-12-03T11:52:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 10 manifest: total_size = 7305 2023-12-03T11:52:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2693513614637591008 2023-12-03T11:52:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.264+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.2 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7305 stripe_ofs=7305 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2693513614637591008 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.t1Ody72CtrHj6WkIyhQizN9LMJ5OpkX_0 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.267+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2693513614637591008 2023-12-03T11:52:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_index_versioning 2023-12-03T11:52:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 10 manifest: total_size = 7644 2023-12-03T11:52:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17074618796293171486 2023-12-03T11:52:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.269+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.2 2023-12-03T11:52:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7644 stripe_ofs=7644 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17074618796293171486 2023-12-03T11:52:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.1dxWJshyR-U6-Av3RLfwHcklVA4-F2N_0 2023-12-03T11:52:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.272+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17074618796293171486 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.273+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_copy 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.273+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.274+0000 7f1d29525800 10 manifest: total_size = 7450 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.274+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15253833040509796658 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.274+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.274+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.274+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.3 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7450 stripe_ofs=7450 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15253833040509796658 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.t_aCElvapfCgA3hayMTTfNuyQ7BJUbp_0 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.276+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15253833040509796658 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_update_complete 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 10 manifest: total_size = 5141 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15979362280409482630 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.278+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.4 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5141 stripe_ofs=5141 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15979362280409482630 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.Loa_SWVcAyVTuviRSYt-7CTqdwJgmHC_0 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.280+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15979362280409482630 2023-12-03T11:52:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_complete 2023-12-03T11:52:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 10 manifest: total_size = 13392 2023-12-03T11:52:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3764564422396261561 2023-12-03T11:52:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.282+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.4 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=13392 stripe_ofs=13392 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3764564422396261561 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.RcpctYp5LKtZFsjuxL5jVxBg8PsKn7L_0 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.285+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3764564422396261561 2023-12-03T11:52:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class 2023-12-03T11:52:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 10 manifest: total_size = 7321 2023-12-03T11:52:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7395227752656352250 2023-12-03T11:52:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.286+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.5 2023-12-03T11:52:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7321 stripe_ofs=7321 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7395227752656352250 2023-12-03T11:52:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.UokA8FBegyNtaR_C1v0ljw7ixbwMH0f_0 2023-12-03T11:52:36.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.289+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7395227752656352250 2023-12-03T11:52:36.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.290+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_simple 2023-12-03T11:52:36.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.290+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.290+0000 7f1d29525800 10 manifest: total_size = 7783 2023-12-03T11:52:36.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.291+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2489060453889625357 2023-12-03T11:52:36.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.291+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.291+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.291+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.7 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7783 stripe_ofs=7783 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2489060453889625357 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.nuPbOHk_FQWz0497UOPBt3ak91Wyoxq_0 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.293+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2489060453889625357 2023-12-03T11:52:36.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart 2023-12-03T11:52:36.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 10 manifest: total_size = 7645 2023-12-03T11:52:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1381388202791168885 2023-12-03T11:52:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x555f4e6aafb8 s->prefetch_data=0 2023-12-03T11:52:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6aafb8 2023-12-03T11:52:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.295+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.3.8 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=7645 stripe_ofs=7645 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1381388202791168885 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.CYb1UMSe9teQYCFEj5H40m28QDm3RDS_0 2023-12-03T11:52:36.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.298+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.13tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.1381388202791168885 2023-12-03T11:52:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.299+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=tests/r_test_small_obj_data 2023-12-03T11:52:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.299+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.299+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.300+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.300+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.300+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.300+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.300+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-conf[21f2df23-11b5-4c91-821f-a8e55156195e.4668.3])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.301+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-conf : miss 2023-12-03T11:52:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.301+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=204 2023-12-03T11:52:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x11 2023-12-03T11:52:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-conf to cache LRU end 2023-12-03T11:52:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-conf from cache 2023-12-03T11:52:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-conf, ofs0, ns] 2023-12-03T11:52:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.302+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660507 cookie 139763068157952 notifier 5777 bl.length()=162 2023-12-03T11:52:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.304+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 from cache 2023-12-03T11:52:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.304+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-conf:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3, ofs0, ns] 2023-12-03T11:52:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.304+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660514 cookie 93867824909968 notifier 5777 bl.length()=219 2023-12-03T11:52:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-conf/21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 : miss 2023-12-03T11:52:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-conf/21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 info.flags=0x0 2023-12-03T11:52:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-conf/21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 to cache LRU end 2023-12-03T11:52:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-conf/21f2df23-11b5-4c91-821f-a8e55156195e.4668.3 from cache 2023-12-03T11:52:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-conf/21f2df23-11b5-4c91-821f-a8e55156195e.4668.3, ofs0, ns] 2023-12-03T11:52:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.308+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660515 cookie 93867824906288 notifier 5777 bl.length()=216 2023-12-03T11:52:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.311+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 : miss 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.311+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=409 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 info.flags=0x17 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 to cache LRU end 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.312+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 2023-12-03T11:52:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 10 manifest: total_size = 74198 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16943246894898149922 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066730 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.315+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.10.10 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=20613 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6312069 part_ofs=20613 rule->part_size=20613 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6332682 part_ofs=41226 rule->part_size=20613 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6353295 part_ofs=61839 rule->part_size=12359 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16943246894898149922 2023-12-03T11:52:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.1 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 347 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.2 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 593 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.3 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 839 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.4 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1085 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.318+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.0tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16943246894898149922 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.319+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.319+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.319+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.321+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.321+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.321+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.321+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.321+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-complete.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.10])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.322+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : miss 2023-12-03T11:52:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.322+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=235 2023-12-03T11:52:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x11 2023-12-03T11:52:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2023-12-03T11:52:36.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 from cache 2023-12-03T11:52:36.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1, ofs0, ns] 2023-12-03T11:52:36.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.323+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660541 cookie 93867822411600 notifier 5777 bl.length()=191 2023-12-03T11:52:36.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.325+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 from cache 2023-12-03T11:52:36.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.325+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10, ofs0, ns] 2023-12-03T11:52:36.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.325+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660508 cookie 93867824908464 notifier 5777 bl.length()=249 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 : miss 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 info.flags=0x0 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 to cache LRU end 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.10 from cache 2023-12-03T11:52:36.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.10, ofs0, ns] 2023-12-03T11:52:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.329+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660542 cookie 93867822411600 notifier 5777 bl.length()=246 2023-12-03T11:52:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 : miss 2023-12-03T11:52:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=416 2023-12-03T11:52:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 info.flags=0x17 2023-12-03T11:52:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 to cache LRU end 2023-12-03T11:52:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.332+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 2023-12-03T11:52:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:52:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18053410 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x555f4e631be8 s->prefetch_data=0 2023-12-03T11:52:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 10 manifest: total_size = 74198 2023-12-03T11:52:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17432873725421458260 2023-12-03T11:52:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18053410 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x555f4e631be8 s->prefetch_data=0 2023-12-03T11:52:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631be8 2023-12-03T11:52:36.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.336+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.11.10 2023-12-03T11:52:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=20613 2023-12-03T11:52:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2023-12-03T11:52:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6312069 part_ofs=20613 rule->part_size=20613 2023-12-03T11:52:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2023-12-03T11:52:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:52:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6332682 part_ofs=41226 rule->part_size=20613 2023-12-03T11:52:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2023-12-03T11:52:36.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6353295 part_ofs=61839 rule->part_size=12359 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17432873725421458260 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.1 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 347 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.2 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 593 2023-12-03T11:52:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.3 2023-12-03T11:52:36.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 839 2023-12-03T11:52:36.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.4 2023-12-03T11:52:36.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1085 2023-12-03T11:52:36.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.339+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17432873725421458260 2023-12-03T11:52:36.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.340+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.340+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.340+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.341+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.341+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.341+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.341+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.341+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-update-complete.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.11])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : miss 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=242 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x11 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 from cache 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.343+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1, ofs0, ns] 2023-12-03T11:52:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.344+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660497 cookie 93867824900608 notifier 5777 bl.length()=198 2023-12-03T11:52:36.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.346+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 from cache 2023-12-03T11:52:36.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.346+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11, ofs0, ns] 2023-12-03T11:52:36.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.346+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660543 cookie 93867822411600 notifier 5777 bl.length()=256 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 : miss 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 info.flags=0x0 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 to cache LRU end 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.11 from cache 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.350+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.11, ofs0, ns] 2023-12-03T11:52:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.351+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660516 cookie 93867824909968 notifier 5777 bl.length()=253 2023-12-03T11:52:36.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.353+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 : miss 2023-12-03T11:52:36.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.353+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=409 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 info.flags=0x17 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 to cache LRU end 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.354+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 2023-12-03T11:52:36.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.357+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo[DuR2pmxPxNO7hFNCMxvQaR8L5e1HsiI] 2023-12-03T11:52:36.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.357+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.358+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x555f4e584d78 s->prefetch_data=0 2023-12-03T11:52:36.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.358+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 8l5o7mltk702n7f4y2gtibmf50w2k98g 2023-12-03T11:52:36.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.358+0000 7f1d29525800 20 get_obj_state_impl: setting s->olh_tag to f04elta51dr0yzal3vy12uo2y2tplsip 2023-12-03T11:52:36.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.358+0000 7f1d29525800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=f04elta51dr0yzal3vy12uo2y2tplsip 2023-12-03T11:52:36.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.359+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.360+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 olh_log_entry: epoch=3 op=2 key=foo[] 2023-12-03T11:52:36.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 olh_log_entry: epoch=3 op=3 key=foo[DuR2pmxPxNO7hFNCMxvQaR8L5e1HsiI] 2023-12-03T11:52:36.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-multipart-index-versioning.1:_:DuR2pmxPxNO7hFNCMxvQaR8L5e1HsiI_foo state=0x555f4e6483e8 s->prefetch_data=0 2023-12-03T11:52:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 10 manifest: total_size = 18874368 2023-12-03T11:52:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3049493997185927048 2023-12-03T11:52:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-multipart-index-versioning.1:_:DuR2pmxPxNO7hFNCMxvQaR8L5e1HsiI_foo state=0x555f4e6483e8 s->prefetch_data=0 2023-12-03T11:52:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6483e8 2023-12-03T11:52:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.361+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=5242880 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=5242880 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=3145728 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=3145728 rules.size()=2 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=22020096 part_ofs=15728640 rule->part_size=3145728 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=18874368 stripe_ofs=18874368 part_ofs=18874368 rule->part_size=3145728 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3049493997185927048 2023-12-03T11:52:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.1 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 344 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.2 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 588 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.3 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 832 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.4 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1076 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.364+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3049493997185927048 2023-12-03T11:52:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.365+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo[DuR2pmxPxNO7hFNCMxvQaR8L5e1HsiI] 2023-12-03T11:52:36.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.367+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x555f4e584d78 s->prefetch_data=0 2023-12-03T11:52:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.369+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.371+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.9.10 2023-12-03T11:52:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.371+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.371+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.372+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.372+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.372+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.372+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.372+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-index-versioning.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.9])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : miss 2023-12-03T11:52:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=235 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x11 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 from cache 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1, ofs0, ns] 2023-12-03T11:52:36.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.374+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660498 cookie 93867824900608 notifier 5777 bl.length()=193 2023-12-03T11:52:36.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.376+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 from cache 2023-12-03T11:52:36.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.376+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9, ofs0, ns] 2023-12-03T11:52:36.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.376+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660499 cookie 93867824900608 notifier 5777 bl.length()=250 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 : miss 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 info.flags=0x0 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 to cache LRU end 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.9 from cache 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.380+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.9, ofs0, ns] 2023-12-03T11:52:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.381+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660544 cookie 93867822411600 notifier 5777 bl.length()=247 2023-12-03T11:52:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.383+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 : miss 2023-12-03T11:52:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.383+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=399 2023-12-03T11:52:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 info.flags=0x17 2023-12-03T11:52:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 to cache LRU end 2023-12-03T11:52:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.384+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 2023-12-03T11:52:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.387+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:52:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.387+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e632fb0 obj=test-client.0-r-test-multipart-simple.1:foo state=0x555f4e585148 s->prefetch_data=0 2023-12-03T11:52:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.388+0000 7f1d29525800 10 manifest: total_size = 18874368 2023-12-03T11:52:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.388+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.8116716281473426768 2023-12-03T11:52:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.388+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e632fb0 obj=test-client.0-r-test-multipart-simple.1:foo state=0x555f4e585148 s->prefetch_data=0 2023-12-03T11:52:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.388+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e585148 2023-12-03T11:52:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.388+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.8.10 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=5242880 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=5242880 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=3145728 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=3145728 rules.size()=2 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=22020096 part_ofs=15728640 rule->part_size=3145728 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=18874368 stripe_ofs=18874368 part_ofs=18874368 rule->part_size=3145728 2023-12-03T11:52:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.8116716281473426768 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.1 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 344 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.2 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 588 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.3 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 832 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.4 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1076 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.390+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.8116716281473426768 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.391+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.391+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.391+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.392+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.392+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.392+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.392+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.392+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-simple.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.8])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.393+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : miss 2023-12-03T11:52:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.393+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.394+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=225 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.394+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x11 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.394+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.394+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 from cache 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.394+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-simple.1, ofs0, ns] 2023-12-03T11:52:36.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.395+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660509 cookie 93867824908464 notifier 5777 bl.length()=183 2023-12-03T11:52:36.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.396+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 from cache 2023-12-03T11:52:36.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.396+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-simple.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8, ofs0, ns] 2023-12-03T11:52:36.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.397+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660510 cookie 93867824908464 notifier 5777 bl.length()=240 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 : miss 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 info.flags=0x0 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 to cache LRU end 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.8 from cache 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.400+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-simple.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.8, ofs0, ns] 2023-12-03T11:52:36.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.401+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660494 cookie 93867824901648 notifier 5777 bl.length()=237 2023-12-03T11:52:36.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.403+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 : miss 2023-12-03T11:52:36.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.403+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.403+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=407 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.403+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 info.flags=0x17 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 to cache LRU end 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 2023-12-03T11:52:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2023-12-03T11:52:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.404+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2023-12-03T11:52:36.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.405+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:52:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.405+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2023-12-03T11:52:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.405+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2023-12-03T11:52:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.405+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:52:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.406+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2023-12-03T11:52:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.406+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:52:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.406+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2023-12-03T11:52:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.407+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2023-12-03T11:52:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.407+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2023-12-03T11:52:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.407+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2023-12-03T11:52:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.407+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e639200 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.408+0000 7f1d29525800 10 manifest: total_size = 2044 2023-12-03T11:52:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.408+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _EXfnec-TpQfJhzhej3L61fCVNxPumD0 2023-12-03T11:52:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.408+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e639200 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.408+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.408+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.0 2023-12-03T11:52:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=2044 stripe_ofs=2044 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _EXfnec-TpQfJhzhej3L61fCVNxPumD0 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.UlJBo2shC-YiPmV44NA7hWVRdT5Mtku_0 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.410+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: _EXfnec-TpQfJhzhej3L61fCVNxPumD0 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.411+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2023-12-03T11:52:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.411+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066d90 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.412+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.412+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _an27X9ERsVYN0rlUmvADL6QHH5Mv4q0 2023-12-03T11:52:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.412+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066d90 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.412+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.412+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.2 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _an27X9ERsVYN0rlUmvADL6QHH5Mv4q0 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._O2YUxZhBMtuQXPCF5im355p1gFn450_0 2023-12-03T11:52:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:52:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.414+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: _an27X9ERsVYN0rlUmvADL6QHH5Mv4q0 2023-12-03T11:52:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.415+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.415+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18058940 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.416+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.416+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4518676667839594598 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.416+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18058940 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.416+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.416+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.3 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4518676667839594598 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_0 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 337 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.418+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4518676667839594598 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.419+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.419+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.420+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.420+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _AV9vx8GvxkaSHP5Akw_NcvEi37-JKQ4 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.420+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.420+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.420+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.5 2023-12-03T11:52:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _AV9vx8GvxkaSHP5Akw_NcvEi37-JKQ4 2023-12-03T11:52:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._F8XjXqNnKGgNXmI8YRqT9YpgP_wG4c_0 2023-12-03T11:52:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:52:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.422+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: _AV9vx8GvxkaSHP5Akw_NcvEi37-JKQ4 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.423+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.423+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18049490 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.424+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.424+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _gv7vn5ZihUg5Jj52XiO7wWpUi7eUvNr 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.424+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18049490 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.424+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.424+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.5 2023-12-03T11:52:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _gv7vn5ZihUg5Jj52XiO7wWpUi7eUvNr 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.dR2zcFiFsWjFQHQZj3d9mEPPTth_dFI_0 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.426+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: _gv7vn5ZihUg5Jj52XiO7wWpUi7eUvNr 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.427+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2023-12-03T11:52:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.427+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066d90 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.428+0000 7f1d29525800 10 manifest: total_size = 2044 2023-12-03T11:52:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.428+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5645470404707769389 2023-12-03T11:52:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.428+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066d90 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.428+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.428+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.5 2023-12-03T11:52:36.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=2044 stripe_ofs=2044 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5645470404707769389 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_0 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 330 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.430+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5645470404707769389 2023-12-03T11:52:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.431+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:52:36.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.431+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.432+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.432+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _eEw5OutN_ftEL2VNNLtwTzFwIRhoZxV 2023-12-03T11:52:36.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.432+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.432+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.432+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.7 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _eEw5OutN_ftEL2VNNLtwTzFwIRhoZxV 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.G__hUT1GN-xsq-p6ifMOPZAeQaSr9uo_0 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:52:36.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.434+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.18tag is: _eEw5OutN_ftEL2VNNLtwTzFwIRhoZxV 2023-12-03T11:52:36.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2023-12-03T11:52:36.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7027472809808533510 2023-12-03T11:52:36.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.435+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.7 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7027472809808533510 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_0 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 339 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.438+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7027472809808533510 2023-12-03T11:52:36.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.439+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:52:36.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 10 manifest: total_size = 2044 2023-12-03T11:52:36.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=2044 stripe_ofs=2044 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_1_5bb6526ccfc1a4edf41357788adf66e4 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_1_5bb6526ccfc1a4edf41357788adf66e4 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.440+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.7 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=2044 stripe_ofs=2044 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_1_5bb6526ccfc1a4edf41357788adf66e4 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_0 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 389 2023-12-03T11:52:36.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.442+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_1_5bb6526ccfc1a4edf41357788adf66e4 2023-12-03T11:52:36.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2023-12-03T11:52:36.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_1_dd6586667d2712fc49bc888a04884fd4 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_1_dd6586667d2712fc49bc888a04884fd4 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.443+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.444+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.8 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_1_dd6586667d2712fc49bc888a04884fd4 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_0 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 398 2023-12-03T11:52:36.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.446+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_1_dd6586667d2712fc49bc888a04884fd4 2023-12-03T11:52:36.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2023-12-03T11:52:36.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 10 manifest: total_size = 2044 2023-12-03T11:52:36.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _cgZJFTbstzVeMT30xjIt9JtgL3AuRIO 2023-12-03T11:52:36.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c083d80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.447+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.8 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=2044 stripe_ofs=2044 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _cgZJFTbstzVeMT30xjIt9JtgL3AuRIO 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._DBUQ0bbe12kEPgYG2zxEdmfGJ4Bvkx_0 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.450+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.17tag is: _cgZJFTbstzVeMT30xjIt9JtgL3AuRIO 2023-12-03T11:52:36.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 10 manifest: total_size = 5242880 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_1_29a088c966b4426474263a92ed1bbae5 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_1_29a088c966b4426474263a92ed1bbae5 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x555f4e637008 s->prefetch_data=0 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e637008 2023-12-03T11:52:36.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.451+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.14.9 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_1_29a088c966b4426474263a92ed1bbae5 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_0 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 396 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.454+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_1_29a088c966b4426474263a92ed1bbae5 2023-12-03T11:52:36.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.455+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2023-12-03T11:52:36.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.455+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.455+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.456+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.456+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.456+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.456+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.456+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-copy.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.14])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.457+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : miss 2023-12-03T11:52:36.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.457+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=233 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x11 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 from cache 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1, ofs0, ns] 2023-12-03T11:52:36.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.458+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660500 cookie 93867824900608 notifier 5777 bl.length()=189 2023-12-03T11:52:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.459+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 from cache 2023-12-03T11:52:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.459+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14, ofs0, ns] 2023-12-03T11:52:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.460+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660535 cookie 93867824920352 notifier 5777 bl.length()=247 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 : miss 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 info.flags=0x0 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 to cache LRU end 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.14 from cache 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.463+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.14, ofs0, ns] 2023-12-03T11:52:36.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.464+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660511 cookie 93867824908464 notifier 5777 bl.length()=244 2023-12-03T11:52:36.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.466+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 : miss 2023-12-03T11:52:36.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.466+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.466+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=417 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.466+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 info.flags=0x17 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.466+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 to cache LRU end 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.467+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.467+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.467+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 2023-12-03T11:52:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.467+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2023-12-03T11:52:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.468+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2023-12-03T11:52:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.468+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:52:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.469+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2023-12-03T11:52:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.470+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2023-12-03T11:52:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.470+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2023-12-03T11:52:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.470+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2023-12-03T11:52:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.470+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e6355b0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.471+0000 7f1d29525800 10 manifest: total_size = 541681 2023-12-03T11:52:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.471+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _W1cRGeeexJOhhMeffN3mBDikz8eor_w 2023-12-03T11:52:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.471+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e6355b0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.471+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.471+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.0 2023-12-03T11:52:36.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _W1cRGeeexJOhhMeffN3mBDikz8eor_w 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.GnBocMFTZPOZhJY6ggTAFLyRjYqfGIQ_0 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.473+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.18tag is: _W1cRGeeexJOhhMeffN3mBDikz8eor_w 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.474+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2023-12-03T11:52:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.474+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.475+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.475+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _lYvrtSJgl4-HEaHIeFnEQMNqtH9-9JY 2023-12-03T11:52:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.475+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.475+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.475+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.2 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _lYvrtSJgl4-HEaHIeFnEQMNqtH9-9JY 2023-12-03T11:52:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.nmchWCxC0NP7hiJT0NJEhWJnJpOYggE_0 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.nmchWCxC0NP7hiJT0NJEhWJnJpOYggE_1 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.477+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: _lYvrtSJgl4-HEaHIeFnEQMNqtH9-9JY 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2791064542133733893 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.478+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.3 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2791064542133733893 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.1 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 367 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 634 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.3 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 901 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.481+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2791064542133733893 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.482+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.482+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.483+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.483+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _evG-0XTX0c6DpgAWHxuk4_q0PeBiVb3 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.483+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.483+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.483+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.5 2023-12-03T11:52:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _evG-0XTX0c6DpgAWHxuk4_q0PeBiVb3 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.Pnv1TDUX2p6_zuRydL7PO2c-zx2pecR_0 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.Pnv1TDUX2p6_zuRydL7PO2c-zx2pecR_1 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 549 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.485+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.31tag is: _evG-0XTX0c6DpgAWHxuk4_q0PeBiVb3 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.486+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.486+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.487+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.487+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _fgeJ7KqvwsZAAvugDm7o6b-2qtDZ3m3 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.487+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.487+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.487+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.5 2023-12-03T11:52:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _fgeJ7KqvwsZAAvugDm7o6b-2qtDZ3m3 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.839ojSGzlETQ10xxfERISGGAiPfrERd_0 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.839ojSGzlETQ10xxfERISGGAiPfrERd_1 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 545 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.489+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.11tag is: _fgeJ7KqvwsZAAvugDm7o6b-2qtDZ3m3 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 10 manifest: total_size = 541681 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14838624973424170250 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.490+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.5 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=246247 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6537703 part_ofs=246247 rule->part_size=246247 2023-12-03T11:52:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6783950 part_ofs=492494 rule->part_size=49187 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14838624973424170250 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.1 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 365 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 629 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.3 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 893 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.493+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.7tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14838624973424170250 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to _zbP6jHYOyjuM6RaZ_dt6BUc5aXQ5ZFu 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.494+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.7 2023-12-03T11:52:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: _zbP6jHYOyjuM6RaZ_dt6BUc5aXQ5ZFu 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.x7P7Zzlo3ljalaSx1Ya3QyyAtgHrZk4_0 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.x7P7Zzlo3ljalaSx1Ya3QyyAtgHrZk4_1 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.498+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _zbP6jHYOyjuM6RaZ_dt6BUc5aXQ5ZFu 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.499+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.499+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.500+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.500+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5277948948846719880 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.500+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.500+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.500+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.7 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5277948948846719880 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.1 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 373 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 646 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.3 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 919 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.502+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.2tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5277948948846719880 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 10 manifest: total_size = 541681 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=246247 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2_384eba532ce0fe32c4793eab7978c65d 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2_384eba532ce0fe32c4793eab7978c65d 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.504+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.7 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=246247 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6537703 part_ofs=246247 rule->part_size=246247 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6783950 part_ofs=492494 rule->part_size=49187 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2_384eba532ce0fe32c4793eab7978c65d 2023-12-03T11:52:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.1 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 440 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 704 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.3 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 968 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.506+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.6tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2_384eba532ce0fe32c4793eab7978c65d 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.507+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c025d90 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2_b5217f9ddabc10ef7ac91bd7af3a8d7b 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2_b5217f9ddabc10ef7ac91bd7af3a8d7b 2023-12-03T11:52:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d1c025d90 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.508+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.8 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2_b5217f9ddabc10ef7ac91bd7af3a8d7b 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.1 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 449 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 722 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.3 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 995 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.510+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.2tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2_b5217f9ddabc10ef7ac91bd7af3a8d7b 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e5fb250 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 10 manifest: total_size = 541681 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to __4_LGsxz9koz2AWclzTLkICdBU2iFvy 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e5fb250 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.512+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.8 2023-12-03T11:52:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: __4_LGsxz9koz2AWclzTLkICdBU2iFvy 2023-12-03T11:52:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.X2qC_WUVOUwsIkECdGXsnOzX2pfxjrp_0 2023-12-03T11:52:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:52:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.515+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: __4_LGsxz9koz2AWclzTLkICdBU2iFvy 2023-12-03T11:52:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.518+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2023-12-03T11:52:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 10 generate_fake_tag new tag=21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2_37bc1b9a1cecd79a1d831445f9312037 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2_37bc1b9a1cecd79a1d831445f9312037 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20004580 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x555f4e631a38 s->prefetch_data=0 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e631a38 2023-12-03T11:52:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.519+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.15.9 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2_37bc1b9a1cecd79a1d831445f9312037 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.1 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 441 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 708 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.3 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 975 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.522+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2_37bc1b9a1cecd79a1d831445f9312037 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.523+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.523+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.523+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.524+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.524+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.524+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.524+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.524+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart-copy.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.15])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : miss 2023-12-03T11:52:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=243 2023-12-03T11:52:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x11 2023-12-03T11:52:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2023-12-03T11:52:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 from cache 2023-12-03T11:52:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1, ofs0, ns] 2023-12-03T11:52:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.526+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660495 cookie 93867824901648 notifier 5777 bl.length()=199 2023-12-03T11:52:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.528+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 from cache 2023-12-03T11:52:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.528+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15, ofs0, ns] 2023-12-03T11:52:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.528+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660501 cookie 93867824900608 notifier 5777 bl.length()=257 2023-12-03T11:52:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 : miss 2023-12-03T11:52:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 info.flags=0x0 2023-12-03T11:52:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 to cache LRU end 2023-12-03T11:52:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.15 from cache 2023-12-03T11:52:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.15, ofs0, ns] 2023-12-03T11:52:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.532+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660536 cookie 93867824920352 notifier 5777 bl.length()=254 2023-12-03T11:52:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.534+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 : miss 2023-12-03T11:52:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.534+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=412 2023-12-03T11:52:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 info.flags=0x17 2023-12-03T11:52:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 to cache LRU end 2023-12-03T11:52:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.535+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 2023-12-03T11:52:36.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.536+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:52:36.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.537+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:52:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.537+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.538+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e585890 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x555f4e64b818 s->prefetch_data=0 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.539+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.539+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2409511898137047588 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.539+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e585890 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x555f4e64b818 s->prefetch_data=0 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.539+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e64b818 2023-12-03T11:52:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.539+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.13.3 2023-12-03T11:52:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2409511898137047588 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.1 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 367 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.2 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 634 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.3 2023-12-03T11:52:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 901 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.541+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2409511898137047588 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.542+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.542+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180616f0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x555f4e64b448 s->prefetch_data=0 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.543+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.543+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5029240034335594067 2023-12-03T11:52:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.543+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180616f0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x555f4e64b448 s->prefetch_data=0 2023-12-03T11:52:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.543+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e64b448 2023-12-03T11:52:36.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.543+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.13.5 2023-12-03T11:52:36.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5029240034335594067 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.1 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 364 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.2 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 628 2023-12-03T11:52:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.3 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 892 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.545+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.13tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5029240034335594067 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.546+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.546+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x555f4e64b448 s->prefetch_data=0 2023-12-03T11:52:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.547+0000 7f1d29525800 10 manifest: total_size = 11534336 2023-12-03T11:52:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.547+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3825308987436127956 2023-12-03T11:52:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.547+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180614e0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x555f4e64b448 s->prefetch_data=0 2023-12-03T11:52:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.547+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e64b448 2023-12-03T11:52:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.547+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.13.7 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3825308987436127956 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.1 2023-12-03T11:52:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 373 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.2 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 646 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.3 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 919 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.549+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.3825308987436127956 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.550+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.551+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.13])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.553+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : miss 2023-12-03T11:52:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.553+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=238 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x11 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 from cache 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1, ofs0, ns] 2023-12-03T11:52:36.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.554+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660517 cookie 93867824906288 notifier 5777 bl.length()=194 2023-12-03T11:52:36.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.555+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 from cache 2023-12-03T11:52:36.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.556+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13, ofs0, ns] 2023-12-03T11:52:36.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.556+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660502 cookie 93867824900608 notifier 5777 bl.length()=252 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.561+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 : miss 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.561+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 info.flags=0x0 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 to cache LRU end 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.13 from cache 2023-12-03T11:52:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.13, ofs0, ns] 2023-12-03T11:52:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.562+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660537 cookie 93867824920352 notifier 5777 bl.length()=249 2023-12-03T11:52:36.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.564+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 : miss 2023-12-03T11:52:36.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.564+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=402 2023-12-03T11:52:36.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 info.flags=0x17 2023-12-03T11:52:36.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 to cache LRU end 2023-12-03T11:52:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.565+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 2023-12-03T11:52:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.566+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:52:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.567+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:52:36.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.568+0000 7f1d29525800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:52:36.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e586200 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 10 manifest: total_size = 10485760 2023-12-03T11:52:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7235507170229646328 2023-12-03T11:52:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e586200 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6394b8 2023-12-03T11:52:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.569+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.12.3 2023-12-03T11:52:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7235507170229646328 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.MNPLcAiQl4YbnSvl82nKaqLqkXtOv1y_0 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 337 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.MNPLcAiQl4YbnSvl82nKaqLqkXtOv1y_1 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 574 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.572+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.7235507170229646328 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a78e0 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 10 manifest: total_size = 44258 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4800958086152152473 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a78e0 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6394b8 2023-12-03T11:52:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.573+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.12.5 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=44258 stripe_ofs=44258 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4800958086152152473 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.Z24apEM09jSDV0bi6uoGQVgG34UlKbJ_0 2023-12-03T11:52:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 330 2023-12-03T11:52:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.576+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4800958086152152473 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 10 manifest: total_size = 10485760 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16633758735362359023 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x555f4e6394b8 s->prefetch_data=0 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6394b8 2023-12-03T11:52:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.579+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.12.7 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16633758735362359023 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.9SYsYG0saUoeEHdHv2f2V9DwlXRseAt_0 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 340 2023-12-03T11:52:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.9SYsYG0saUoeEHdHv2f2V9DwlXRseAt_1 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 579 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.582+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.16633758735362359023 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.583+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.12])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : miss 2023-12-03T11:52:36.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=228 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x11 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 from cache 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.585+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1, ofs0, ns] 2023-12-03T11:52:36.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.586+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660503 cookie 93867824900608 notifier 5777 bl.length()=184 2023-12-03T11:52:36.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.587+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 from cache 2023-12-03T11:52:36.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.587+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12, ofs0, ns] 2023-12-03T11:52:36.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.587+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660545 cookie 93867822411600 notifier 5777 bl.length()=242 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 : miss 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 info.flags=0x0 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 to cache LRU end 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.12 from cache 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.12, ofs0, ns] 2023-12-03T11:52:36.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.591+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660518 cookie 93867824909968 notifier 5777 bl.length()=239 2023-12-03T11:52:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 : miss 2023-12-03T11:52:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 info.flags=0x17 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 to cache LRU end 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.594+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 2023-12-03T11:52:36.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.595+0000 7f1d29525800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:52:36.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.596+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:52:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.597+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:52:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.597+0000 7f1d29525800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:52:36.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e62b0b0 obj=test-client.0-r-test-small-obj-data.1:___ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 10 manifest: total_size = 0 2023-12-03T11:52:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14837917363434854520 2023-12-03T11:52:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e62b0b0 obj=test-client.0-r-test-small-obj-data.1:___ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6560c8 2023-12-03T11:52:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.598+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.5.2 2023-12-03T11:52:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a7740 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 10 manifest: total_size = 0 2023-12-03T11:52:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.9627048389434814175 2023-12-03T11:52:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a7740 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6560c8 2023-12-03T11:52:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.601+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.5.3 2023-12-03T11:52:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.602+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:52:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180615d0 obj=test-client.0-r-test-small-obj-data.1:__ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 10 manifest: total_size = 0 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14933640205400274883 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180615d0 obj=test-client.0-r-test-small-obj-data.1:__ state=0x555f4e6560c8 s->prefetch_data=0 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e6560c8 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.604+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.5.6 2023-12-03T11:52:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.605+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:52:36.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a7740 obj=test-client.0-r-test-small-obj-data.1:obj state=0x555f4e651e88 s->prefetch_data=0 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 10 manifest: total_size = 0 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.14210990181635624014 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a7740 obj=test-client.0-r-test-small-obj-data.1:obj state=0x555f4e651e88 s->prefetch_data=0 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e651e88 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.607+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.5.8 2023-12-03T11:52:36.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.608+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:52:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.610+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.610+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:52:36.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.611+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.1[21f2df23-11b5-4c91-821f-a8e55156195e.4668.5])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : miss 2023-12-03T11:52:36.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x11 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 from cache 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.1, ofs0, ns] 2023-12-03T11:52:36.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.613+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660546 cookie 93867822411600 notifier 5777 bl.length()=181 2023-12-03T11:52:36.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.615+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 from cache 2023-12-03T11:52:36.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.615+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.1:21f2df23-11b5-4c91-821f-a8e55156195e.4668.5, ofs0, ns] 2023-12-03T11:52:36.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.615+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660547 cookie 93867822411600 notifier 5777 bl.length()=238 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 : miss 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 info.flags=0x0 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 to cache LRU end 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.5 from cache 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.620+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.1/21f2df23-11b5-4c91-821f-a8e55156195e.4668.5, ofs0, ns] 2023-12-03T11:52:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.621+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660512 cookie 139763068157952 notifier 5777 bl.length()=235 2023-12-03T11:52:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.623+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 : miss 2023-12-03T11:52:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.623+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 info.flags=0x17 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 to cache LRU end 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.624+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 2023-12-03T11:52:36.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.625+0000 7f1d29525800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:52:36.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.625+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:52:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.626+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:52:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.627+0000 7f1d29525800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:52:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.627+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e638400 obj=test-client.0-r-test-small-obj-data.2:___ state=0x555f4e63b658 s->prefetch_data=0 2023-12-03T11:52:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.628+0000 7f1d29525800 10 manifest: total_size = 38 2023-12-03T11:52:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.628+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17044074510798205480 2023-12-03T11:52:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.628+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e638400 obj=test-client.0-r-test-small-obj-data.2:___ state=0x555f4e63b658 s->prefetch_data=0 2023-12-03T11:52:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.628+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e63b658 2023-12-03T11:52:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.628+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.6.2 2023-12-03T11:52:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=38 stripe_ofs=38 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17044074510798205480 2023-12-03T11:52:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.HyYv3r4rIXbbhoRKwMLNfakRHPM_qhE_0 2023-12-03T11:52:36.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.630+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.17044074510798205480 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.631+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x555f4e63b658 s->prefetch_data=0 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.631+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.632+0000 7f1d29525800 10 manifest: total_size = 524288 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.632+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4771697587165576162 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.632+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x555f4e63b658 s->prefetch_data=0 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.632+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e63b658 2023-12-03T11:52:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.632+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.6.3 2023-12-03T11:52:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=524288 stripe_ofs=524288 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4771697587165576162 2023-12-03T11:52:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.LNNaVLlCDXO5Xh4QvVIyVM4LqS9prsA_0 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.634+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.9tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.4771697587165576162 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.635+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:52:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.635+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180615d0 obj=test-client.0-r-test-small-obj-data.2:__ state=0x555f4e63aa38 s->prefetch_data=0 2023-12-03T11:52:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.636+0000 7f1d29525800 10 manifest: total_size = 38 2023-12-03T11:52:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.636+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.456617908150793397 2023-12-03T11:52:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.636+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180615d0 obj=test-client.0-r-test-small-obj-data.2:__ state=0x555f4e63aa38 s->prefetch_data=0 2023-12-03T11:52:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.636+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e63aa38 2023-12-03T11:52:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.636+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.6.6 2023-12-03T11:52:36.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=38 stripe_ofs=38 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.456617908150793397 2023-12-03T11:52:36.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.Ie5LuIlHKy0Et9w2FhG1PMMQeMZxNBw_0 2023-12-03T11:52:36.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 327 2023-12-03T11:52:36.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.638+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.456617908150793397 2023-12-03T11:52:36.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:52:36.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-small-obj-data.2:obj state=0x555f4e63aa38 s->prefetch_data=0 2023-12-03T11:52:36.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 10 manifest: total_size = 38 2023-12-03T11:52:36.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15437882696372543921 2023-12-03T11:52:36.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d18088bc0 obj=test-client.0-r-test-small-obj-data.2:obj state=0x555f4e63aa38 s->prefetch_data=0 2023-12-03T11:52:36.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e63aa38 2023-12-03T11:52:36.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.639+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.6.8 2023-12-03T11:52:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=38 stripe_ofs=38 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15437882696372543921 2023-12-03T11:52:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.mVeNYCiMspEMDAHlkZ02aOgz9YNTaj3_0 2023-12-03T11:52:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.642+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.20tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15437882696372543921 2023-12-03T11:52:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.643+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:52:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.643+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.643+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.644+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.644+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.644+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.644+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.644+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.2[21f2df23-11b5-4c91-821f-a8e55156195e.4668.6])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : miss 2023-12-03T11:52:36.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x11 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 from cache 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.2, ofs0, ns] 2023-12-03T11:52:36.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.646+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660548 cookie 93867822411600 notifier 5777 bl.length()=181 2023-12-03T11:52:36.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.648+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 from cache 2023-12-03T11:52:36.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.648+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.2:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6, ofs0, ns] 2023-12-03T11:52:36.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.648+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660549 cookie 93867822411600 notifier 5777 bl.length()=238 2023-12-03T11:52:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 : miss 2023-12-03T11:52:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 info.flags=0x0 2023-12-03T11:52:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 to cache LRU end 2023-12-03T11:52:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/21f2df23-11b5-4c91-821f-a8e55156195e.4668.6 from cache 2023-12-03T11:52:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.2/21f2df23-11b5-4c91-821f-a8e55156195e.4668.6, ofs0, ns] 2023-12-03T11:52:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.652+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660513 cookie 139763068157952 notifier 5777 bl.length()=235 2023-12-03T11:52:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.655+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 : miss 2023-12-03T11:52:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.655+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 info.flags=0x17 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 to cache LRU end 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:52:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 2023-12-03T11:52:36.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.656+0000 7f1d29525800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:52:36.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.657+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:52:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.658+0000 7f1d29525800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:52:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.658+0000 7f1d29525800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:52:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066470 obj=test-client.0-r-test-small-obj-data.3:___ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 10 manifest: total_size = 1035 2023-12-03T11:52:36.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2351895489632245315 2023-12-03T11:52:36.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d20066470 obj=test-client.0-r-test-small-obj-data.3:___ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e639d18 2023-12-03T11:52:36.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.659+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.7.2 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=1035 stripe_ofs=1035 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2351895489632245315 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.6-JA0l1lh4phThqUZ_vQXabD8nDLGLK_0 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.662+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.29tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.2351895489632245315 2023-12-03T11:52:36.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:52:36.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e639a80 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 10 manifest: total_size = 1048576 2023-12-03T11:52:36.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15921661692034423210 2023-12-03T11:52:36.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 20 get_obj_state: octx=0x555f4e639a80 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e639d18 2023-12-03T11:52:36.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.663+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.7.3 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=1048576 stripe_ofs=1048576 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15921661692034423210 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.0Z13W0BHcscL3D7gZI-Bj1_DFZ2OXY4_0 2023-12-03T11:52:36.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:52:36.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.666+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.15921661692034423210 2023-12-03T11:52:36.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:52:36.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180616f0 obj=test-client.0-r-test-small-obj-data.3:__ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d29525800 10 manifest: total_size = 1035 2023-12-03T11:52:36.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.321573323563310060 2023-12-03T11:52:36.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d180616f0 obj=test-client.0-r-test-small-obj-data.3:__ state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.667+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e639d18 2023-12-03T11:52:36.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.668+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.7.6 2023-12-03T11:52:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=1035 stripe_ofs=1035 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.321573323563310060 2023-12-03T11:52:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.-1tlZPjr8xlamemArrtRxoCZMtoiCmB_0 2023-12-03T11:52:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 327 2023-12-03T11:52:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.670+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.321573323563310060 2023-12-03T11:52:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.671+0000 7f1d077fe640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:52:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.671+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a78e0 obj=test-client.0-r-test-small-obj-data.3:obj state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.672+0000 7f1d29525800 10 manifest: total_size = 1035 2023-12-03T11:52:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.672+0000 7f1d29525800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.6062969768876879931 2023-12-03T11:52:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.672+0000 7f1d29525800 20 get_obj_state: octx=0x7f1d200a78e0 obj=test-client.0-r-test-small-obj-data.3:obj state=0x555f4e639d18 s->prefetch_data=0 2023-12-03T11:52:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.672+0000 7f1d29525800 20 prepare_atomic_modification: state is not atomic. state=0x555f4e639d18 2023-12-03T11:52:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.672+0000 7f1d29525800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.7.8 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 RGWObjManifest::operator++(): result: ofs=1035 stripe_ofs=1035 part_ofs=0 rule->part_size=0 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.6062969768876879931 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.84b-a49TxSI-75YaRNe343zGi1uEtqH_0 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:52:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.674+0000 7f1d29525800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: 21f2df23-11b5-4c91-821f-a8e55156195e.4662.6062969768876879931 2023-12-03T11:52:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d06ffd640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:52:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:52:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-12-03T11:52:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:52:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:52:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:52:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:52:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.676+0000 7f1d29525800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.3[21f2df23-11b5-4c91-821f-a8e55156195e.4668.7])"; uploads.size()=0; is_truncated=0 2023-12-03T11:52:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.678+0000 7f1d29525800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : miss 2023-12-03T11:52:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.678+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d29525800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d29525800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x11 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d29525800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d29525800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 from cache 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.3, ofs0, ns] 2023-12-03T11:52:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.679+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660550 cookie 93867822411600 notifier 5777 bl.length()=181 2023-12-03T11:52:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.680+0000 7f1d29525800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 from cache 2023-12-03T11:52:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.680+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.3:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7, ofs0, ns] 2023-12-03T11:52:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.680+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660551 cookie 93867822411600 notifier 5777 bl.length()=238 2023-12-03T11:52:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 : miss 2023-12-03T11:52:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 info.flags=0x0 2023-12-03T11:52:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 to cache LRU end 2023-12-03T11:52:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/21f2df23-11b5-4c91-821f-a8e55156195e.4668.7 from cache 2023-12-03T11:52:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.3/21f2df23-11b5-4c91-821f-a8e55156195e.4668.7, ofs0, ns] 2023-12-03T11:52:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.684+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660514 cookie 139763068157952 notifier 5777 bl.length()=235 2023-12-03T11:52:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.687+0000 7f1d29525800 10 removing key index: UTBOVHAHAGVNNTWYJBFD 2023-12-03T11:52:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.687+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:UTBOVHAHAGVNNTWYJBFD, ofs0, ns] 2023-12-03T11:52:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.687+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660504 cookie 93867824900608 notifier 5777 bl.length()=170 2023-12-03T11:52:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.688+0000 7f1d29525800 10 removing email index: ragweed.client.0+test@test.test 2023-12-03T11:52:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.688+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2023-12-03T11:52:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.689+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660505 cookie 93867824900608 notifier 5777 bl.length()=182 2023-12-03T11:52:36.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.690+0000 7f1d29525800 10 removing user buckets index 2023-12-03T11:52:36.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.690+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0.buckets, ofs0, ns] 2023-12-03T11:52:36.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.690+0000 7f1d077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660519 cookie 93867824906288 notifier 5777 bl.length()=173 2023-12-03T11:52:36.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.692+0000 7f1d29525800 10 removing user index: ragweed.client.0 2023-12-03T11:52:36.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.692+0000 7f1d29525800 10 removing default.rgw.meta+users.uid+ragweed.client.0 from cache 2023-12-03T11:52:36.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.692+0000 7f1d29525800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2023-12-03T11:52:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.692+0000 7f1d06ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660515 cookie 93867824908464 notifier 5777 bl.length()=165 2023-12-03T11:52:36.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.694+0000 7f1ceafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.696+0000 7f1d29525800 20 remove_watcher() i=0 2023-12-03T11:52:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.696+0000 7f1d29525800 2 removed watcher, disabling cache 2023-12-03T11:52:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.698+0000 7f1d29525800 20 remove_watcher() i=1 2023-12-03T11:52:36.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.699+0000 7f1d29525800 20 remove_watcher() i=2 2023-12-03T11:52:36.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.700+0000 7f1d29525800 20 remove_watcher() i=3 2023-12-03T11:52:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.701+0000 7f1d29525800 20 remove_watcher() i=4 2023-12-03T11:52:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.702+0000 7f1d29525800 20 remove_watcher() i=5 2023-12-03T11:52:36.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.703+0000 7f1d29525800 20 remove_watcher() i=6 2023-12-03T11:52:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.704+0000 7f1d29525800 20 remove_watcher() i=7 2023-12-03T11:52:36.710 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid sysuser.client.0 --purge-data 2023-12-03T11:52:36.793 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:52:36.793 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:52:36.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.803+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.804+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.804+0000 7f22573af800 20 realm 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.804+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.805+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.805+0000 7f22573af800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:52:36.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.805+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.805+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.805+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.806+0000 7f22573af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.806+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.806+0000 7f22573af800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:52:36.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.806+0000 7f22573af800 20 searching for the correct realm 2023-12-03T11:52:36.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.815+0000 7f224ffff640 0 -- 172.21.15.116:0/3017693044 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x555a14f93400 msgr2=0x555a14f957f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:36.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:52:36.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:52:36.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:52:36.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:52:36.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.816+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 zone default found 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 4 Realm: () 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:52:36.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 zonegroup default 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 started sync module instance, tier type = 2023-12-03T11:52:36.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.817+0000 7f22573af800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:52:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.824+0000 7f224ffff640 0 -- 172.21.15.116:0/3017693044 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7f2230079e10 msgr2=0x7f223007c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=0 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=1 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=2 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=3 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=4 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=5 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.826+0000 7f22573af800 20 add_watcher() i=6 2023-12-03T11:52:36.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.827+0000 7f22573af800 20 add_watcher() i=7 2023-12-03T11:52:36.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.827+0000 7f22573af800 2 all 8 watchers are set, enabling cache 2023-12-03T11:52:36.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.828+0000 7f221a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:36.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.829+0000 7f22573af800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:52:36.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:36.829+0000 7f22573af800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:52:37.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.070+0000 7f21897fa640 20 reqs_thread_entry: start 2023-12-03T11:52:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.071+0000 7f22573af800 20 init_complete bucket index max shards: 11 2023-12-03T11:52:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.071+0000 7f22573af800 20 Filter name: none 2023-12-03T11:52:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.071+0000 7f2182ffd640 20 reqs_thread_entry: start 2023-12-03T11:52:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.071+0000 7f22573af800 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2023-12-03T11:52:37.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.071+0000 7f22573af800 20 rados->read ofs=0 len=0 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 20 rados_obj.operate() r=0 bl.length=452 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x13 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 10 removing key index: ABIGYRFREBRMBKJXURTF 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f22573af800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:ABIGYRFREBRMBKJXURTF, ofs0, ns] 2023-12-03T11:52:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.072+0000 7f2236ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660520 cookie 93845387319456 notifier 5795 bl.length()=170 2023-12-03T11:52:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.074+0000 7f22573af800 10 removing email index: sysuser.client.0+test@test.test 2023-12-03T11:52:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.074+0000 7f22573af800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2023-12-03T11:52:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.074+0000 7f22367fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660516 cookie 93845387299920 notifier 5795 bl.length()=182 2023-12-03T11:52:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.076+0000 7f22573af800 10 removing user buckets index 2023-12-03T11:52:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.076+0000 7f22573af800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.uid:sysuser.client.0.buckets, ofs0, ns] 2023-12-03T11:52:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.076+0000 7f2236ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660496 cookie 93845387313184 notifier 5795 bl.length()=173 2023-12-03T11:52:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.077+0000 7f22573af800 10 removing user index: sysuser.client.0 2023-12-03T11:52:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.077+0000 7f22573af800 10 removing default.rgw.meta+users.uid+sysuser.client.0 from cache 2023-12-03T11:52:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.077+0000 7f22573af800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2023-12-03T11:52:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.077+0000 7f22367fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660506 cookie 93845387308880 notifier 5795 bl.length()=165 2023-12-03T11:52:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.080+0000 7f221a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:52:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.082+0000 7f22573af800 20 remove_watcher() i=0 2023-12-03T11:52:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.082+0000 7f22573af800 2 removed watcher, disabling cache 2023-12-03T11:52:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.083+0000 7f22573af800 20 remove_watcher() i=1 2023-12-03T11:52:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.084+0000 7f22573af800 20 remove_watcher() i=2 2023-12-03T11:52:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.085+0000 7f22573af800 20 remove_watcher() i=3 2023-12-03T11:52:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.086+0000 7f22573af800 20 remove_watcher() i=4 2023-12-03T11:52:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.086+0000 7f22573af800 20 remove_watcher() i=5 2023-12-03T11:52:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.087+0000 7f22573af800 20 remove_watcher() i=6 2023-12-03T11:52:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:52:37.088+0000 7f22573af800 20 remove_watcher() i=7 2023-12-03T11:52:37.095 INFO:tasks.ragweed:Removing ragweed... 2023-12-03T11:52:37.095 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:52:37.192 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:52:37.203 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:52:37.203 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:52:37.203 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:52:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:52:37.218 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:52:37.218 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:52:37.272 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:52:37.273 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:52:37.326 INFO:tasks.workunit:timeout=3h 2023-12-03T11:52:37.326 INFO:tasks.workunit:cleanup=True 2023-12-03T11:52:37.327 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:52:37.381 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:53:42.046 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.047 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:53:42.048 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:53:42.049 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:53:42.104 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:53:42.106 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:53:42.106 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:53:42.155 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:53:42.204 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:53:42.243 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:53:42.244 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:53:42.245 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:53:42.283 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:53:42.286 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:53:42.286 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:53:42.342 INFO:tasks.workunit:Running workunits matching rgw/run-reshard.sh on client.0... 2023-12-03T11:53:42.343 INFO:tasks.workunit:Running workunit rgw/run-reshard.sh... 2023-12-03T11:53:42.343 DEBUG:teuthology.orchestra.run.smithi116:workunit test rgw/run-reshard.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2023-12-03T11:53:42.400 INFO:tasks.workunit.client.0.smithi116.stderr:+ ulimit -c 0 2023-12-03T11:53:42.400 INFO:tasks.workunit.client.0.smithi116.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2023-12-03T11:53:42.401 INFO:tasks.workunit.client.0.smithi116.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:53:42.401 INFO:tasks.workunit.client.0.smithi116.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:53:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2023-12-03T11:53:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:++ deactivate nondestructive 2023-12-03T11:53:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:53:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:53:44.769 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ hash -r 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ unset VIRTUAL_ENV 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' '!' nondestructive = nondestructive ']' 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ export VIRTUAL_ENV 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ _OLD_VIRTUAL_PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ export PATH 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' ']' 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -z '' ']' 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ _OLD_VIRTUAL_PS1= 2023-12-03T11:53:44.770 INFO:tasks.workunit.client.0.smithi116.stderr:++ PS1='(rgw) ' 2023-12-03T11:53:44.771 INFO:tasks.workunit.client.0.smithi116.stderr:++ export PS1 2023-12-03T11:53:44.771 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:53:44.771 INFO:tasks.workunit.client.0.smithi116.stderr:++ hash -r 2023-12-03T11:53:44.771 INFO:tasks.workunit.client.0.smithi116.stderr:+ pip install pip --upgrade 2023-12-03T11:53:44.997 INFO:tasks.workunit.client.0.smithi116.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:53:45.236 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting pip 2023-12-03T11:53:45.241 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:53:45.288 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: pip 2023-12-03T11:53:45.288 INFO:tasks.workunit.client.0.smithi116.stdout: Attempting uninstall: pip 2023-12-03T11:53:45.288 INFO:tasks.workunit.client.0.smithi116.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:53:45.437 INFO:tasks.workunit.client.0.smithi116.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:53:45.443 INFO:tasks.workunit.client.0.smithi116.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:53:46.532 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed pip-23.3.1 2023-12-03T11:53:46.578 INFO:tasks.workunit.client.0.smithi116.stderr:+ pip install boto3 2023-12-03T11:53:47.379 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting boto3 2023-12-03T11:53:47.380 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:53:47.888 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:53:47.889 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:53:47.940 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:53:47.941 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:53:47.994 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:53:47.996 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:53:48.053 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:53:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:53:48.145 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:53:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:53:48.301 INFO:tasks.workunit.client.0.smithi116.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:53:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:53:48.325 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:53:48.326 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:53:48.346 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:53:48.347 INFO:tasks.workunit.client.0.smithi116.stdout:Using cached urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:53:48.445 INFO:tasks.workunit.client.0.smithi116.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:53:49.492 INFO:tasks.workunit.client.0.smithi116.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:53:49.614 INFO:tasks.workunit.client.0.smithi116.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_reshard.py 2023-12-03T11:53:49.737 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin user create --uid tester --display-name "Testing" --access-key NX5QOQKC6BH2IDN8HC7A --secret LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn 2023-12-03T11:53:49.770 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:53:49.770 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:53:49.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.782+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 realm 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.784+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.785+0000 7f1d0f016800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:53:49.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.785+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.786+0000 7f1d0f016800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:53:49.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.786+0000 7f1d0f016800 20 searching for the correct realm 2023-12-03T11:53:49.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.796+0000 7f1d0a693640 0 -- 172.21.15.116:0/3318824376 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55c796d61a60 msgr2=0x55c796d63e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:53:49.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.797+0000 7f1d0f016800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.797+0000 7f1d0f016800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.797+0000 7f1d0f016800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.797+0000 7f1d0f016800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.797+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:53:49.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:53:49.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 zone default found 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 4 Realm: () 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 zonegroup default 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.798+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.799+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.799+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.799+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.799+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.800+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.800+0000 7f1d0f016800 20 started sync module instance, tier type = 2023-12-03T11:53:49.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.800+0000 7f1d0f016800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:53:49.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.807+0000 7f1d03fff640 0 -- 172.21.15.116:0/3318824376 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55c796d669a0 msgr2=0x55c796d68d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:53:49.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=0 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=1 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=2 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=3 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=4 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=5 2023-12-03T11:53:49.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.810+0000 7f1d0f016800 20 add_watcher() i=6 2023-12-03T11:53:49.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.811+0000 7f1d0f016800 20 add_watcher() i=7 2023-12-03T11:53:49.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.811+0000 7f1d0f016800 2 all 8 watchers are set, enabling cache 2023-12-03T11:53:49.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.813+0000 7f1ccffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:53:49.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.813+0000 7f1d0f016800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:53:49.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.813+0000 7f1d0f016800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1d0f016800 5 note: GC not initialized 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1c3effd640 20 reqs_thread_entry: start 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1d0f016800 20 init_complete bucket index max shards: 11 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1d0f016800 20 Filter name: none 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1c3cff9640 20 reqs_thread_entry: start 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1d0f016800 10 cache get: name=default.rgw.meta+users.uid+tester : miss 2023-12-03T11:53:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.819+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 adding default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : miss 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 20 rados->read ofs=0 len=0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x0 2023-12-03T11:53:49.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:53:49.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-12-03T11:53:49.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.820+0000 7f1d0f016800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-12-03T11:53:49.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.821+0000 7f1d0f016800 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2023-12-03T11:53:49.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.821+0000 7f1d0f016800 10 moving default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:53:49.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.821+0000 7f1d0f016800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester, ofs0, ns] 2023-12-03T11:53:49.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.822+0000 7f1d00ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660521 cookie 94315717488560 notifier 6126 bl.length()=535 2023-12-03T11:53:49.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.822+0000 7f1d00ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2023-12-03T11:53:49.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.822+0000 7f1d00ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d0f016800 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d0f016800 10 moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d0f016800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d00ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660522 cookie 94315717506864 notifier 6126 bl.length()=180 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d00ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-12-03T11:53:49.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.823+0000 7f1d00ff9640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:53:49.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.824+0000 7f1ccffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:53:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.827+0000 7f1d0f016800 20 remove_watcher() i=0 2023-12-03T11:53:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.827+0000 7f1d0f016800 2 removed watcher, disabling cache 2023-12-03T11:53:49.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.828+0000 7f1d0f016800 20 remove_watcher() i=1 2023-12-03T11:53:49.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.829+0000 7f1d0f016800 20 remove_watcher() i=2 2023-12-03T11:53:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.830+0000 7f1d0f016800 20 remove_watcher() i=3 2023-12-03T11:53:49.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.831+0000 7f1d0f016800 20 remove_watcher() i=4 2023-12-03T11:53:49.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.832+0000 7f1d0f016800 20 remove_watcher() i=5 2023-12-03T11:53:49.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.833+0000 7f1d0f016800 20 remove_watcher() i=6 2023-12-03T11:53:49.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:53:49.834+0000 7f1d0f016800 20 remove_watcher() i=7 2023-12-03T11:54:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.763 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.809 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.824 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.837 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.842 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:06.886 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard add 2023-12-03T11:54:06.927 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:06.929 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:54:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:06.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.970+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 realm 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.972+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.973+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:06.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.973+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.973+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:06.975 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.973+0000 7fab1c923800 20 searching for the correct realm 2023-12-03T11:54:06.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.984+0000 7fab1853c640 0 -- 172.21.15.116:0/4057425604 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x565498bd5280 msgr2=0x565498bd7670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:06.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.985+0000 7fab1c923800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:06.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.985+0000 7fab1c923800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:06.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.985+0000 7fab1c923800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:06.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.985+0000 7fab1c923800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:06.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.985+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 zone default found 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 4 Realm: () 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 zonegroup default 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.986+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 started sync module instance, tier type = 2023-12-03T11:54:06.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.987+0000 7fab1c923800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:06.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.994+0000 7fab0fd3b640 0 -- 172.21.15.116:0/4057425604 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x565498bd9e50 msgr2=0x565498bdc2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:06.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=0 2023-12-03T11:54:06.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=1 2023-12-03T11:54:06.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=2 2023-12-03T11:54:06.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=3 2023-12-03T11:54:06.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=4 2023-12-03T11:54:06.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=5 2023-12-03T11:54:06.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.997+0000 7fab1c923800 20 add_watcher() i=6 2023-12-03T11:54:06.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.998+0000 7fab1c923800 20 add_watcher() i=7 2023-12-03T11:54:06.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.998+0000 7fab1c923800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:06.999+0000 7faade7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.000+0000 7fab1c923800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.000+0000 7fab1c923800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.005+0000 7fab1c923800 5 note: GC not initialized 2023-12-03T11:54:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7faa4d7fa640 20 reqs_thread_entry: start 2023-12-03T11:54:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7fab1c923800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7fab1c923800 20 Filter name: none 2023-12-03T11:54:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7faa46ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.006+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.007+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.007+0000 7fab1c923800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.007+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.007+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.007+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.008+0000 7fab1c923800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.008+0000 7fab1c923800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.009+0000 7fab1c923800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.009+0000 7fab1c923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.010+0000 7faade7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.012+0000 7fab1c923800 20 remove_watcher() i=0 2023-12-03T11:54:07.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.012+0000 7fab1c923800 2 removed watcher, disabling cache 2023-12-03T11:54:07.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.013+0000 7fab1c923800 20 remove_watcher() i=1 2023-12-03T11:54:07.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.014+0000 7fab1c923800 20 remove_watcher() i=2 2023-12-03T11:54:07.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.015+0000 7fab1c923800 20 remove_watcher() i=3 2023-12-03T11:54:07.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.016+0000 7fab1c923800 20 remove_watcher() i=4 2023-12-03T11:54:07.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.017+0000 7fab1c923800 20 remove_watcher() i=5 2023-12-03T11:54:07.019 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.018+0000 7fab1c923800 20 remove_watcher() i=6 2023-12-03T11:54:07.020 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.019+0000 7fab1c923800 20 remove_watcher() i=7 2023-12-03T11:54:07.025 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:07.025 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:07.025 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: a-bucket id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 num_objects: 0 size_kb: 0 num_shards: 11 2023-12-03T11:54:07.025 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:07.026 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 12 2023-12-03T11:54:07.055 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:07.055 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:07.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.068+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 realm 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.069+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.070+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.070+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.071+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.071+0000 7fc521b31800 20 searching for the correct realm 2023-12-03T11:54:07.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.081+0000 7fc51d83c640 0 -- 172.21.15.116:0/4024019927 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5558ef6976d0 msgr2=0x5558ef699ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.083+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 zone default found 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 4 Realm: () 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 zonegroup default 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 started sync module instance, tier type = 2023-12-03T11:54:07.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.084+0000 7fc521b31800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:07.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.091+0000 7fc51e03d640 0 -- 172.21.15.116:0/4024019927 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fc514076180 msgr2=0x7fc514076c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=0 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=1 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=2 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=3 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=4 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=5 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.093+0000 7fc521b31800 20 add_watcher() i=6 2023-12-03T11:54:07.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.094+0000 7fc521b31800 20 add_watcher() i=7 2023-12-03T11:54:07.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.094+0000 7fc521b31800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:07.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.096+0000 7fc4e37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.096+0000 7fc521b31800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:07.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.096+0000 7fc521b31800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:07.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc521b31800 5 note: GC not initialized 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc4527fc640 20 reqs_thread_entry: start 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc521b31800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc521b31800 20 Filter name: none 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc443fff640 20 reqs_thread_entry: start 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc521b31800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:54:07.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.102+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:54:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:54:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : miss 2023-12-03T11:54:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.103+0000 7fc521b31800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:07.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.105+0000 7fc4e37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.107+0000 7fc521b31800 20 remove_watcher() i=0 2023-12-03T11:54:07.109 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.107+0000 7fc521b31800 2 removed watcher, disabling cache 2023-12-03T11:54:07.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.108+0000 7fc521b31800 20 remove_watcher() i=1 2023-12-03T11:54:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.109+0000 7fc521b31800 20 remove_watcher() i=2 2023-12-03T11:54:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.110+0000 7fc521b31800 20 remove_watcher() i=3 2023-12-03T11:54:07.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.111+0000 7fc521b31800 20 remove_watcher() i=4 2023-12-03T11:54:07.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.112+0000 7fc521b31800 20 remove_watcher() i=5 2023-12-03T11:54:07.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.113+0000 7fc521b31800 20 remove_watcher() i=6 2023-12-03T11:54:07.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.114+0000 7fc521b31800 20 remove_watcher() i=7 2023-12-03T11:54:07.122 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard list 2023-12-03T11:54:07.147 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:07.147 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:07.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.159+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 20 realm 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.161+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.162+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.162+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.162+0000 7f8916d23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.162+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.163+0000 7f8916d23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:07.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.163+0000 7f8916d23800 20 searching for the correct realm 2023-12-03T11:54:07.173 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.172+0000 7f8912943640 0 -- 172.21.15.116:0/4080451734 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55595dce6ac0 msgr2=0x55595dce8eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.173+0000 7f8916d23800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.173+0000 7f8916d23800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.173+0000 7f8916d23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.173+0000 7f8916d23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.173+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 zone default found 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 4 Realm: () 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 zonegroup default 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.174+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.175+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.175+0000 7f8916d23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.175+0000 7f8916d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.175+0000 7f8916d23800 20 started sync module instance, tier type = 2023-12-03T11:54:07.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.175+0000 7f8916d23800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:07.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.181+0000 7f8913144640 0 -- 172.21.15.116:0/4080451734 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f890805acc0 msgr2=0x7f8908059e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:07.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=0 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=1 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=2 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=3 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=4 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=5 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.184+0000 7f8916d23800 20 add_watcher() i=6 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.185+0000 7f8916d23800 20 add_watcher() i=7 2023-12-03T11:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.185+0000 7f8916d23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:07.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.186+0000 7f8853fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.187+0000 7f8916d23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:07.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.187+0000 7f8916d23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:07.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.192+0000 7f8916d23800 5 note: GC not initialized 2023-12-03T11:54:07.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.193+0000 7f884affd640 20 reqs_thread_entry: start 2023-12-03T11:54:07.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.193+0000 7f8916d23800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:07.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.193+0000 7f8916d23800 20 Filter name: none 2023-12-03T11:54:07.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.193+0000 7f8848ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:07.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.201+0000 7f8853fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.203+0000 7f8916d23800 20 remove_watcher() i=0 2023-12-03T11:54:07.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.203+0000 7f8916d23800 2 removed watcher, disabling cache 2023-12-03T11:54:07.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.204+0000 7f8916d23800 20 remove_watcher() i=1 2023-12-03T11:54:07.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.205+0000 7f8916d23800 20 remove_watcher() i=2 2023-12-03T11:54:07.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.206+0000 7f8916d23800 20 remove_watcher() i=3 2023-12-03T11:54:07.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.207+0000 7f8916d23800 20 remove_watcher() i=4 2023-12-03T11:54:07.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.208+0000 7f8916d23800 20 remove_watcher() i=5 2023-12-03T11:54:07.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.209+0000 7f8916d23800 20 remove_watcher() i=6 2023-12-03T11:54:07.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.210+0000 7f8916d23800 20 remove_watcher() i=7 2023-12-03T11:54:07.216 INFO:tasks.workunit.client.0.smithi116.stderr:bucket name a-bucket 2023-12-03T11:54:07.216 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard process 2023-12-03T11:54:07.216 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:07.217 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard process 2023-12-03T11:54:07.242 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:07.242 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:07.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.255+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 20 realm 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.256+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.257+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.257+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.257+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.257+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.258+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:07.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.258+0000 7f4275ddc800 20 searching for the correct realm 2023-12-03T11:54:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.267+0000 7f42738dc640 0 -- 172.21.15.116:0/1195714844 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55dd57ca5140 msgr2=0x55dd57ca7530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.268+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 zone default found 2023-12-03T11:54:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 4 Realm: () 2023-12-03T11:54:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 zonegroup default 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 started sync module instance, tier type = 2023-12-03T11:54:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.269+0000 7f4275ddc800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:07.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.277+0000 7f426b0db640 0 -- 172.21.15.116:0/1195714844 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55dd57caa170 msgr2=0x55dd57cac560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:07.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=0 2023-12-03T11:54:07.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=1 2023-12-03T11:54:07.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=2 2023-12-03T11:54:07.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=3 2023-12-03T11:54:07.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=4 2023-12-03T11:54:07.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=5 2023-12-03T11:54:07.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.278+0000 7f4275ddc800 20 add_watcher() i=6 2023-12-03T11:54:07.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.279+0000 7f4275ddc800 20 add_watcher() i=7 2023-12-03T11:54:07.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.279+0000 7f4275ddc800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.281+0000 7f4239fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.281+0000 7f4275ddc800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.281+0000 7f4275ddc800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:07.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f4275ddc800 5 note: GC not initialized 2023-12-03T11:54:07.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f41b0ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:07.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f4275ddc800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:07.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f4275ddc800 20 Filter name: none 2023-12-03T11:54:07.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f41a67fc640 20 reqs_thread_entry: start 2023-12-03T11:54:07.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.287+0000 7f4275ddc800 20 processing logshard = reshard.0000000000 2023-12-03T11:54:07.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.290+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000000 , ret = 0 2023-12-03T11:54:07.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.290+0000 7f4275ddc800 20 processing logshard = reshard.0000000001 2023-12-03T11:54:07.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.292+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000001 , ret = 0 2023-12-03T11:54:07.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.292+0000 7f4275ddc800 20 processing logshard = reshard.0000000002 2023-12-03T11:54:07.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.295+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000002 , ret = 0 2023-12-03T11:54:07.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.295+0000 7f4275ddc800 20 processing logshard = reshard.0000000003 2023-12-03T11:54:07.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.298+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000003 , ret = 0 2023-12-03T11:54:07.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.298+0000 7f4275ddc800 20 processing logshard = reshard.0000000004 2023-12-03T11:54:07.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.300+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000004 , ret = 0 2023-12-03T11:54:07.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.300+0000 7f4275ddc800 20 processing logshard = reshard.0000000005 2023-12-03T11:54:07.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.303+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000005 , ret = 0 2023-12-03T11:54:07.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.303+0000 7f4275ddc800 20 processing logshard = reshard.0000000006 2023-12-03T11:54:07.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 20 process_entry resharding a-bucket 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : miss 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 20 rados->read ofs=0 len=0 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.305+0000 7f4275ddc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f426a0d9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660519 cookie 94409149052048 notifier 5897 bl.length()=789 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f426a0d9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f426a0d9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f426a0d9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:54:07.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:54:07.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.311+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:54:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.0 2023-12-03T11:54:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.0 2023-12-03T11:54:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.1 2023-12-03T11:54:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.1 2023-12-03T11:54:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.2 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.2 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.3 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.3 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.4 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.4 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.5 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.5 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.6 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.6 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.7 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.7 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.8 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.8 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.9 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.9 2023-12-03T11:54:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.10 2023-12-03T11:54:07.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.10 2023-12-03T11:54:07.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.11 2023-12-03T11:54:07.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.11 2023-12-03T11:54:07.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.313+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.0 2023-12-03T11:54:07.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.314+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.314+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.315+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.3 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.315+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.4 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.315+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.5 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.316+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.6 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.316+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.7 2023-12-03T11:54:07.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.317+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.8 2023-12-03T11:54:07.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.317+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.9 2023-12-03T11:54:07.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.318+0000 7f4275ddc800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.10 2023-12-03T11:54:07.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.318+0000 7f4275ddc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:07.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.318+0000 7f4275ddc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.319+0000 7f4275ddc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.319+0000 7f4275ddc800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.319+0000 7f4275ddc800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.319+0000 7f4275ddc800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f426a0d9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660520 cookie 94409149052048 notifier 5897 bl.length()=757 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f426a0d9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f426a0d9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f426a0d9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:54:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:54:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:54:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.320+0000 7f4275ddc800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:54:07.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.322+0000 7f4275ddc800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:54:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.323+0000 7f4275ddc800 20 process_entry removing reshard queue entry for bucket a-bucket 2023-12-03T11:54:07.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.325+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000006 , ret = 0 2023-12-03T11:54:07.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.325+0000 7f4275ddc800 20 processing logshard = reshard.0000000007 2023-12-03T11:54:07.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.327+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000007 , ret = 0 2023-12-03T11:54:07.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.327+0000 7f4275ddc800 20 processing logshard = reshard.0000000008 2023-12-03T11:54:07.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.330+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000008 , ret = 0 2023-12-03T11:54:07.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.330+0000 7f4275ddc800 20 processing logshard = reshard.0000000009 2023-12-03T11:54:07.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.332+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000009 , ret = 0 2023-12-03T11:54:07.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.332+0000 7f4275ddc800 20 processing logshard = reshard.0000000010 2023-12-03T11:54:07.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.334+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000010 , ret = 0 2023-12-03T11:54:07.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.334+0000 7f4275ddc800 20 processing logshard = reshard.0000000011 2023-12-03T11:54:07.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.336+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000011 , ret = 0 2023-12-03T11:54:07.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.336+0000 7f4275ddc800 20 processing logshard = reshard.0000000012 2023-12-03T11:54:07.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.338+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000012 , ret = 0 2023-12-03T11:54:07.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.338+0000 7f4275ddc800 20 processing logshard = reshard.0000000013 2023-12-03T11:54:07.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.341+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000013 , ret = 0 2023-12-03T11:54:07.342 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.341+0000 7f4275ddc800 20 processing logshard = reshard.0000000014 2023-12-03T11:54:07.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.343+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000014 , ret = 0 2023-12-03T11:54:07.344 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.343+0000 7f4275ddc800 20 processing logshard = reshard.0000000015 2023-12-03T11:54:07.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.345+0000 7f4275ddc800 20 finish processing logshard = reshard.0000000015 , ret = 0 2023-12-03T11:54:07.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.346+0000 7f4239fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:07.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.348+0000 7f4275ddc800 20 remove_watcher() i=0 2023-12-03T11:54:07.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.348+0000 7f4275ddc800 2 removed watcher, disabling cache 2023-12-03T11:54:07.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.349+0000 7f4275ddc800 20 remove_watcher() i=1 2023-12-03T11:54:07.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.350+0000 7f4275ddc800 20 remove_watcher() i=2 2023-12-03T11:54:07.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.351+0000 7f4275ddc800 20 remove_watcher() i=3 2023-12-03T11:54:07.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.352+0000 7f4275ddc800 20 remove_watcher() i=4 2023-12-03T11:54:07.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.353+0000 7f4275ddc800 20 remove_watcher() i=5 2023-12-03T11:54:07.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.354+0000 7f4275ddc800 20 remove_watcher() i=6 2023-12-03T11:54:07.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:07.355+0000 7f4275ddc800 20 remove_watcher() i=7 2023-12-03T11:54:12.368 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:54:12.399 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.399 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.413+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 realm 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.415+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.416+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.416+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.417+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.417+0000 7fbaecb26800 20 searching for the correct realm 2023-12-03T11:54:12.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.428+0000 7fbae8723640 0 -- 172.21.15.116:0/2156071574 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fbae005e050 msgr2=0x7fbae005f430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.429+0000 7fbaecb26800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.429+0000 7fbaecb26800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.429+0000 7fbaecb26800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.429+0000 7fbaecb26800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.429+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.432 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 zone default found 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 4 Realm: () 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 zonegroup default 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.430+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 started sync module instance, tier type = 2023-12-03T11:54:12.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.431+0000 7fbaecb26800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.437+0000 7fbae8723640 0 -- 172.21.15.116:0/2156071574 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fbac0079e40 msgr2=0x7fbac007c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=0 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=1 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=2 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=3 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=4 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=5 2023-12-03T11:54:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.441+0000 7fbaecb26800 20 add_watcher() i=6 2023-12-03T11:54:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.442+0000 7fbaecb26800 20 add_watcher() i=7 2023-12-03T11:54:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.442+0000 7fbaecb26800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:12.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.444+0000 7fbaae7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.444+0000 7fbaecb26800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:12.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.444+0000 7fbaecb26800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:12.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fbaecb26800 5 note: GC not initialized 2023-12-03T11:54:12.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fba1d7fa640 20 reqs_thread_entry: start 2023-12-03T11:54:12.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fbaecb26800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:12.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fbaecb26800 20 Filter name: none 2023-12-03T11:54:12.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fba16ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:12.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.451+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.452+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.452+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.452+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.452+0000 7fbaecb26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.452+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.453+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.453+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.453+0000 7fbaecb26800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.453+0000 7fbaecb26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.454+0000 7fbaecb26800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.454+0000 7fbaecb26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.455+0000 7fbaae7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.457+0000 7fbaecb26800 20 remove_watcher() i=0 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.457+0000 7fbaecb26800 2 removed watcher, disabling cache 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.458+0000 7fbaecb26800 20 remove_watcher() i=1 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.460+0000 7fbaecb26800 20 remove_watcher() i=2 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.461+0000 7fbaecb26800 20 remove_watcher() i=3 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.462+0000 7fbaecb26800 20 remove_watcher() i=4 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.463+0000 7fbaecb26800 20 remove_watcher() i=5 2023-12-03T11:54:12.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.464+0000 7fbaecb26800 20 remove_watcher() i=6 2023-12-03T11:54:12.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.465+0000 7fbaecb26800 20 remove_watcher() i=7 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: a-bucket id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 num_objects: 0 size_kb: 0 num_shards: 12 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard add non empty bucket 2023-12-03T11:54:12.471 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.476 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.485 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.494 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.502 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.510 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.517 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.525 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:12.533 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:12.540 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:54:12.566 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.566 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:12.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.578+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 20 realm 2023-12-03T11:54:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.580+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.581+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.581+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.581+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.581+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.582+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:12.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.582+0000 7f95dc144800 20 searching for the correct realm 2023-12-03T11:54:12.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.591+0000 7f95d7f0a640 0 -- 172.21.15.116:0/4012967053 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55ff47e7a990 msgr2=0x55ff47e7cd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.592+0000 7f95dc144800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.592+0000 7f95dc144800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.592+0000 7f95dc144800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.592+0000 7f95dc144800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.592+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 zone default found 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 4 Realm: () 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 zonegroup default 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.593+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.594+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.594+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.594+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.594+0000 7f95dc144800 20 started sync module instance, tier type = 2023-12-03T11:54:12.597 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.594+0000 7f95dc144800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:12.602 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.600+0000 7f95d7709640 0 -- 172.21.15.116:0/4012967053 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55ff47e7f900 msgr2=0x55ff47e81cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=0 2023-12-03T11:54:12.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=1 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=2 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=3 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=4 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=5 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.603+0000 7f95dc144800 20 add_watcher() i=6 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.604+0000 7f95dc144800 20 add_watcher() i=7 2023-12-03T11:54:12.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.604+0000 7f95dc144800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:12.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.606+0000 7f959dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.606+0000 7f95dc144800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:12.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.606+0000 7f95dc144800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:12.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.611+0000 7f95dc144800 5 note: GC not initialized 2023-12-03T11:54:12.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f9514ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:12.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f95dc144800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:12.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f95dc144800 20 Filter name: none 2023-12-03T11:54:12.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f950a7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:12.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.612+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.613+0000 7f95dc144800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:12.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.614+0000 7f95dc144800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.614+0000 7f95dc144800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.615+0000 7f959dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.617+0000 7f95dc144800 20 remove_watcher() i=0 2023-12-03T11:54:12.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.617+0000 7f95dc144800 2 removed watcher, disabling cache 2023-12-03T11:54:12.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.618+0000 7f95dc144800 20 remove_watcher() i=1 2023-12-03T11:54:12.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.619+0000 7f95dc144800 20 remove_watcher() i=2 2023-12-03T11:54:12.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.620+0000 7f95dc144800 20 remove_watcher() i=3 2023-12-03T11:54:12.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.621+0000 7f95dc144800 20 remove_watcher() i=4 2023-12-03T11:54:12.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.622+0000 7f95dc144800 20 remove_watcher() i=5 2023-12-03T11:54:12.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.623+0000 7f95dc144800 20 remove_watcher() i=6 2023-12-03T11:54:12.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.624+0000 7f95dc144800 20 remove_watcher() i=7 2023-12-03T11:54:12.630 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.630 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:12.630 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: a-bucket id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 num_objects: 8 size_kb: 1 num_shards: 12 2023-12-03T11:54:12.630 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.631 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 13 2023-12-03T11:54:12.660 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.660 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:12.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.672+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.673+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.673+0000 7f9f19812800 20 realm 2023-12-03T11:54:12.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.673+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.674+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.674+0000 7f9f19812800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.674+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.674+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.674+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.675+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.675+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.675+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.675+0000 7f9f19812800 20 searching for the correct realm 2023-12-03T11:54:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.684+0000 7f9f0ffff640 0 -- 172.21.15.116:0/1711619812 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5593c9ed81f0 msgr2=0x5593c9eda5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.685+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 zone default found 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 4 Realm: () 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 zonegroup default 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 started sync module instance, tier type = 2023-12-03T11:54:12.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.686+0000 7f9f19812800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:12.695 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.694+0000 7f9f0ffff640 0 -- 172.21.15.116:0/1711619812 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7f9ef0079e10 msgr2=0x7f9ef007c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=0 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=1 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=2 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=3 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=4 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=5 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.695+0000 7f9f19812800 20 add_watcher() i=6 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.696+0000 7f9f19812800 20 add_watcher() i=7 2023-12-03T11:54:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.696+0000 7f9f19812800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:12.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.697+0000 7f9eda7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.698+0000 7f9f19812800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:12.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.698+0000 7f9f19812800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:12.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 5 note: GC not initialized 2023-12-03T11:54:12.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9e497fa640 20 reqs_thread_entry: start 2023-12-03T11:54:12.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:12.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 20 Filter name: none 2023-12-03T11:54:12.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9e42ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:12.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:54:12.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : miss 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.704+0000 7f9f19812800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.705+0000 7f9f19812800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.707+0000 7f9eda7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.709+0000 7f9f19812800 20 remove_watcher() i=0 2023-12-03T11:54:12.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.709+0000 7f9f19812800 2 removed watcher, disabling cache 2023-12-03T11:54:12.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.710+0000 7f9f19812800 20 remove_watcher() i=1 2023-12-03T11:54:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.711+0000 7f9f19812800 20 remove_watcher() i=2 2023-12-03T11:54:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.712+0000 7f9f19812800 20 remove_watcher() i=3 2023-12-03T11:54:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.713+0000 7f9f19812800 20 remove_watcher() i=4 2023-12-03T11:54:12.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.714+0000 7f9f19812800 20 remove_watcher() i=5 2023-12-03T11:54:12.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.715+0000 7f9f19812800 20 remove_watcher() i=6 2023-12-03T11:54:12.718 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.716+0000 7f9f19812800 20 remove_watcher() i=7 2023-12-03T11:54:12.723 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard list 2023-12-03T11:54:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.749 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:12.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.761+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 realm 2023-12-03T11:54:12.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.763+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.764+0000 7f5317b3d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.764+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.764+0000 7f5317b3d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:12.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.764+0000 7f5317b3d800 20 searching for the correct realm 2023-12-03T11:54:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f531409d640 0 -- 172.21.15.116:0/1720167885 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f52fc032910 msgr2=0x7f52fc021390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.774+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 zone default found 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 4 Realm: () 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 zonegroup default 2023-12-03T11:54:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.775+0000 7f5317b3d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.776+0000 7f5317b3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.776+0000 7f5317b3d800 20 started sync module instance, tier type = 2023-12-03T11:54:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.776+0000 7f5317b3d800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f531389c640 0 -- 172.21.15.116:0/1720167885 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f52fc0233b0 msgr2=0x7f52fc023830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=0 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=1 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=2 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=3 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=4 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=5 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.784+0000 7f5317b3d800 20 add_watcher() i=6 2023-12-03T11:54:12.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.785+0000 7f5317b3d800 20 add_watcher() i=7 2023-12-03T11:54:12.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.785+0000 7f5317b3d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:12.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.787+0000 7f52d9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.787+0000 7f5317b3d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:12.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.787+0000 7f5317b3d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:12.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.793+0000 7f5317b3d800 5 note: GC not initialized 2023-12-03T11:54:12.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.793+0000 7f5250ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:12.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.793+0000 7f5317b3d800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:12.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.793+0000 7f5317b3d800 20 Filter name: none 2023-12-03T11:54:12.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.793+0000 7f52467fc640 20 reqs_thread_entry: start 2023-12-03T11:54:12.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.801+0000 7f52d9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.803+0000 7f5317b3d800 20 remove_watcher() i=0 2023-12-03T11:54:12.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.803+0000 7f5317b3d800 2 removed watcher, disabling cache 2023-12-03T11:54:12.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.804+0000 7f5317b3d800 20 remove_watcher() i=1 2023-12-03T11:54:12.806 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.805+0000 7f5317b3d800 20 remove_watcher() i=2 2023-12-03T11:54:12.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.806+0000 7f5317b3d800 20 remove_watcher() i=3 2023-12-03T11:54:12.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.807+0000 7f5317b3d800 20 remove_watcher() i=4 2023-12-03T11:54:12.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.807+0000 7f5317b3d800 20 remove_watcher() i=5 2023-12-03T11:54:12.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.808+0000 7f5317b3d800 20 remove_watcher() i=6 2023-12-03T11:54:12.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.809+0000 7f5317b3d800 20 remove_watcher() i=7 2023-12-03T11:54:12.815 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard process non empty bucket 2023-12-03T11:54:12.815 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:12.817 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin reshard process 2023-12-03T11:54:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:12.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.855+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 realm 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.856+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.857+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.857+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.857+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:12.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.857+0000 7f64e6335800 20 searching for the correct realm 2023-12-03T11:54:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.864+0000 7f64e2870640 0 -- 172.21.15.116:0/1490788760 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f64cc017030 msgr2=0x7f64cc0566e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.868 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.867+0000 7f64e206f640 0 -- 172.21.15.116:0/1490788760 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x5603ee3d5d50 msgr2=0x7f64dc026940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.868+0000 7f64e6335800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.868+0000 7f64e6335800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.868+0000 7f64e6335800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.868+0000 7f64e6335800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.868+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.870 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 zone default found 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 4 Realm: () 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.869+0000 7f64e6335800 20 zonegroup default 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.872 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 started sync module instance, tier type = 2023-12-03T11:54:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.870+0000 7f64e6335800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:12.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=0 2023-12-03T11:54:12.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=1 2023-12-03T11:54:12.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=2 2023-12-03T11:54:12.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=3 2023-12-03T11:54:12.880 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=4 2023-12-03T11:54:12.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=5 2023-12-03T11:54:12.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.879+0000 7f64e6335800 20 add_watcher() i=6 2023-12-03T11:54:12.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.880+0000 7f64e6335800 20 add_watcher() i=7 2023-12-03T11:54:12.881 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.880+0000 7f64e6335800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:12.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.882+0000 7f64a7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.883+0000 7f64e6335800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:12.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.883+0000 7f64e6335800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.889+0000 7f64e6335800 5 note: GC not initialized 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.889+0000 7f6416ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.889+0000 7f64e6335800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.889+0000 7f64e6335800 20 Filter name: none 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.890+0000 7f6414ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:12.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.890+0000 7f64e6335800 20 processing logshard = reshard.0000000000 2023-12-03T11:54:12.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.892+0000 7f64e6335800 20 finish processing logshard = reshard.0000000000 , ret = 0 2023-12-03T11:54:12.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.892+0000 7f64e6335800 20 processing logshard = reshard.0000000001 2023-12-03T11:54:12.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.895+0000 7f64e6335800 20 finish processing logshard = reshard.0000000001 , ret = 0 2023-12-03T11:54:12.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.895+0000 7f64e6335800 20 processing logshard = reshard.0000000002 2023-12-03T11:54:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.898+0000 7f64e6335800 20 finish processing logshard = reshard.0000000002 , ret = 0 2023-12-03T11:54:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.898+0000 7f64e6335800 20 processing logshard = reshard.0000000003 2023-12-03T11:54:12.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.900+0000 7f64e6335800 20 finish processing logshard = reshard.0000000003 , ret = 0 2023-12-03T11:54:12.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.900+0000 7f64e6335800 20 processing logshard = reshard.0000000004 2023-12-03T11:54:12.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.902+0000 7f64e6335800 20 finish processing logshard = reshard.0000000004 , ret = 0 2023-12-03T11:54:12.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.902+0000 7f64e6335800 20 processing logshard = reshard.0000000005 2023-12-03T11:54:12.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.905+0000 7f64e6335800 20 finish processing logshard = reshard.0000000005 , ret = 0 2023-12-03T11:54:12.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.905+0000 7f64e6335800 20 processing logshard = reshard.0000000006 2023-12-03T11:54:12.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.906+0000 7f64e6335800 20 process_entry resharding a-bucket 2023-12-03T11:54:12.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.906+0000 7f64e6335800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.906+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : miss 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 20 rados->read ofs=0 len=0 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.907+0000 7f64e6335800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:12.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64d886c640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660521 cookie 94574883561056 notifier 6315 bl.length()=789 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64d886c640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64d886c640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64d886c640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:54:12.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.914+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:54:12.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:54:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:54:12.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.0 2023-12-03T11:54:12.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.0 2023-12-03T11:54:12.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.1 2023-12-03T11:54:12.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.917+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.1 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.2 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.2 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.3 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.3 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.4 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.918+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.4 2023-12-03T11:54:12.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.919+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.5 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.919+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.5 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.919+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.6 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.919+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.6 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.7 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.7 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.8 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.8 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.9 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.920+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.9 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.921+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.10 2023-12-03T11:54:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.921+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.10 2023-12-03T11:54:12.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.921+0000 7f64e6335800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.11 2023-12-03T11:54:12.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.921+0000 7f64e6335800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.1.11 2023-12-03T11:54:12.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.924+0000 7f64e6335800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:12.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.924+0000 7f64e6335800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:54:12.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:54:12.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64c3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660522 cookie 94574883561056 notifier 6315 bl.length()=757 2023-12-03T11:54:12.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64c3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:54:12.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64c3fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:54:12.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64c3fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:54:12.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:54:12.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:54:12.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:54:12.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.925+0000 7f64e6335800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:54:12.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.928+0000 7f64e6335800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:54:12.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.929+0000 7f64e6335800 20 process_entry removing reshard queue entry for bucket a-bucket 2023-12-03T11:54:12.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.931+0000 7f64e6335800 20 finish processing logshard = reshard.0000000006 , ret = 0 2023-12-03T11:54:12.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.931+0000 7f64e6335800 20 processing logshard = reshard.0000000007 2023-12-03T11:54:12.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.933+0000 7f64e6335800 20 finish processing logshard = reshard.0000000007 , ret = 0 2023-12-03T11:54:12.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.933+0000 7f64e6335800 20 processing logshard = reshard.0000000008 2023-12-03T11:54:12.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.935+0000 7f64e6335800 20 finish processing logshard = reshard.0000000008 , ret = 0 2023-12-03T11:54:12.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.935+0000 7f64e6335800 20 processing logshard = reshard.0000000009 2023-12-03T11:54:12.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.937+0000 7f64e6335800 20 finish processing logshard = reshard.0000000009 , ret = 0 2023-12-03T11:54:12.939 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.937+0000 7f64e6335800 20 processing logshard = reshard.0000000010 2023-12-03T11:54:12.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.940+0000 7f64e6335800 20 finish processing logshard = reshard.0000000010 , ret = 0 2023-12-03T11:54:12.941 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.940+0000 7f64e6335800 20 processing logshard = reshard.0000000011 2023-12-03T11:54:12.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.942+0000 7f64e6335800 20 finish processing logshard = reshard.0000000011 , ret = 0 2023-12-03T11:54:12.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.942+0000 7f64e6335800 20 processing logshard = reshard.0000000012 2023-12-03T11:54:12.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.945+0000 7f64e6335800 20 finish processing logshard = reshard.0000000012 , ret = 0 2023-12-03T11:54:12.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.945+0000 7f64e6335800 20 processing logshard = reshard.0000000013 2023-12-03T11:54:12.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.947+0000 7f64e6335800 20 finish processing logshard = reshard.0000000013 , ret = 0 2023-12-03T11:54:12.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.947+0000 7f64e6335800 20 processing logshard = reshard.0000000014 2023-12-03T11:54:12.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.949+0000 7f64e6335800 20 finish processing logshard = reshard.0000000014 , ret = 0 2023-12-03T11:54:12.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.949+0000 7f64e6335800 20 processing logshard = reshard.0000000015 2023-12-03T11:54:12.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.952+0000 7f64e6335800 20 finish processing logshard = reshard.0000000015 , ret = 0 2023-12-03T11:54:12.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.952+0000 7f64a7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:12.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.954+0000 7f64e6335800 20 remove_watcher() i=0 2023-12-03T11:54:12.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.954+0000 7f64e6335800 2 removed watcher, disabling cache 2023-12-03T11:54:12.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.955+0000 7f64e6335800 20 remove_watcher() i=1 2023-12-03T11:54:12.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.957+0000 7f64e6335800 20 remove_watcher() i=2 2023-12-03T11:54:12.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.958+0000 7f64e6335800 20 remove_watcher() i=3 2023-12-03T11:54:12.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.959+0000 7f64e6335800 20 remove_watcher() i=4 2023-12-03T11:54:12.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.960+0000 7f64e6335800 20 remove_watcher() i=5 2023-12-03T11:54:12.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.960+0000 7f64e6335800 20 remove_watcher() i=6 2023-12-03T11:54:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:12.961+0000 7f64e6335800 20 remove_watcher() i=7 2023-12-03T11:54:12.969 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:54:12.996 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:12.996 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.008+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.009+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.009+0000 7f40f493d800 20 realm 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.009+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.010+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.011+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.011+0000 7f40f493d800 20 searching for the correct realm 2023-12-03T11:54:13.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.020+0000 7f40f06a3640 0 -- 172.21.15.116:0/3820847730 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x561019918ea0 msgr2=0x56101991b290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.021+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 zone default found 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 4 Realm: () 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 zonegroup default 2023-12-03T11:54:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.022+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.023+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.023+0000 7f40f493d800 20 started sync module instance, tier type = 2023-12-03T11:54:13.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.023+0000 7f40f493d800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.028+0000 7f40e7ea2640 0 -- 172.21.15.116:0/3820847730 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x56101991dab0 msgr2=0x56101991ff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=0 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=1 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=2 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=3 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=4 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=5 2023-12-03T11:54:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.031+0000 7f40f493d800 20 add_watcher() i=6 2023-12-03T11:54:13.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.032+0000 7f40f493d800 20 add_watcher() i=7 2023-12-03T11:54:13.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.032+0000 7f40f493d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.034+0000 7f40b67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.034+0000 7f40f493d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.034+0000 7f40f493d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f40f493d800 5 note: GC not initialized 2023-12-03T11:54:13.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f40257fa640 20 reqs_thread_entry: start 2023-12-03T11:54:13.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f40f493d800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f40f493d800 20 Filter name: none 2023-12-03T11:54:13.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f401effd640 20 reqs_thread_entry: start 2023-12-03T11:54:13.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.040+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.041+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.042+0000 7f40f493d800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:54:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.042+0000 7f40f493d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.043+0000 7f40f493d800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.043+0000 7f40f493d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.043+0000 7f40b67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.045+0000 7f40f493d800 20 remove_watcher() i=0 2023-12-03T11:54:13.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.045+0000 7f40f493d800 2 removed watcher, disabling cache 2023-12-03T11:54:13.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.046+0000 7f40f493d800 20 remove_watcher() i=1 2023-12-03T11:54:13.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.047+0000 7f40f493d800 20 remove_watcher() i=2 2023-12-03T11:54:13.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.048+0000 7f40f493d800 20 remove_watcher() i=3 2023-12-03T11:54:13.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.049+0000 7f40f493d800 20 remove_watcher() i=4 2023-12-03T11:54:13.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.050+0000 7f40f493d800 20 remove_watcher() i=5 2023-12-03T11:54:13.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.051+0000 7f40f493d800 20 remove_watcher() i=6 2023-12-03T11:54:13.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.052+0000 7f40f493d800 20 remove_watcher() i=7 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: a-bucket id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 num_objects: 8 size_kb: 1 num_shards: 13 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with EIO injected at set_target_layout 2023-12-03T11:54:13.058 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.060 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.072 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.116 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.124 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.133 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.141 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.148 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.156 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.164 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.171 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.179 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.186 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.194 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.202 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.210 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.217 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.225 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.233 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.241 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.248 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.256 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.264 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.272 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:54:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.310+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 realm 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.312+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.313+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.313+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.313+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.313+0000 7fd162f1e800 20 searching for the correct realm 2023-12-03T11:54:13.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.323+0000 7fd160224640 0 -- 172.21.15.116:0/3691341773 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x564828f8b920 msgr2=0x564828f8dd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 zone default found 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 4 Realm: () 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 zonegroup default 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.324+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 started sync module instance, tier type = 2023-12-03T11:54:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.325+0000 7fd162f1e800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.332+0000 7fd160224640 0 -- 172.21.15.116:0/3691341773 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7fd138079e10 msgr2=0x7fd13807c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=0 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=1 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=2 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=3 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=4 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=5 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.334+0000 7fd162f1e800 20 add_watcher() i=6 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.335+0000 7fd162f1e800 20 add_watcher() i=7 2023-12-03T11:54:13.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.335+0000 7fd162f1e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.337+0000 7fd1277de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.338+0000 7fd162f1e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.338+0000 7fd162f1e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.540+0000 7fd160224640 0 --2- 172.21.15.116:0/303172508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564828fbd390 0x564828fd0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:54:13.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd162f1e800 5 note: GC not initialized 2023-12-03T11:54:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd0967fc640 20 reqs_thread_entry: start 2023-12-03T11:54:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd162f1e800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd162f1e800 20 Filter name: none 2023-12-03T11:54:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd087fff640 20 reqs_thread_entry: start 2023-12-03T11:54:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.545+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.546+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.547+0000 7fd162f1e800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.547+0000 7fd162f1e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.548+0000 7fd162f1e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.548+0000 7fd162f1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.549+0000 7fd1277de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.551+0000 7fd162f1e800 20 remove_watcher() i=0 2023-12-03T11:54:13.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.551+0000 7fd162f1e800 2 removed watcher, disabling cache 2023-12-03T11:54:13.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.552+0000 7fd162f1e800 20 remove_watcher() i=1 2023-12-03T11:54:13.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.553+0000 7fd162f1e800 20 remove_watcher() i=2 2023-12-03T11:54:13.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.554+0000 7fd162f1e800 20 remove_watcher() i=3 2023-12-03T11:54:13.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.555+0000 7fd162f1e800 20 remove_watcher() i=4 2023-12-03T11:54:13.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.556+0000 7fd162f1e800 20 remove_watcher() i=5 2023-12-03T11:54:13.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.557+0000 7fd162f1e800 20 remove_watcher() i=6 2023-12-03T11:54:13.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.558+0000 7fd162f1e800 20 remove_watcher() i=7 2023-12-03T11:54:13.564 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.564 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:13.564 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:13.564 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.565 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout 2023-12-03T11:54:13.597 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:13.597 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.609+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 realm 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.611+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.612+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.612+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.612+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.614 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.612+0000 7fe6f51af800 20 searching for the correct realm 2023-12-03T11:54:13.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.622+0000 7fe6f2caf640 0 -- 172.21.15.116:0/2012749008 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5579b2e5f020 msgr2=0x5579b2e61430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 zone default found 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 4 Realm: () 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 zonegroup default 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.623+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.626 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 started sync module instance, tier type = 2023-12-03T11:54:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.624+0000 7fe6f51af800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.631+0000 7fe6ea4ae640 0 -- 172.21.15.116:0/2012749008 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5579b2e64020 msgr2=0x5579b2e664c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=0 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=1 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=2 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=3 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=4 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=5 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.633+0000 7fe6f51af800 20 add_watcher() i=6 2023-12-03T11:54:13.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.634+0000 7fe6f51af800 20 add_watcher() i=7 2023-12-03T11:54:13.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.634+0000 7fe6f51af800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.636+0000 7fe633fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.636+0000 7fe6f51af800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.636+0000 7fe6f51af800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe6f51af800 5 note: GC not initialized 2023-12-03T11:54:13.643 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe622ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:13.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe6f51af800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe6f51af800 20 Filter name: none 2023-12-03T11:54:13.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe620ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:13.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe6f51af800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:54:13.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.642+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 : miss 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.643+0000 7fe6f51af800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 2023-12-03T11:54:13.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.647+0000 7fe6f51af800 -1 Injecting error=-5 at location=set_target_layout 2023-12-03T11:54:13.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.647+0000 7fe6f51af800 0 ERROR: init_target_layout failed to write target index layout to bucket info: (5) Input/output error 2023-12-03T11:54:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.651+0000 7fe633fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.653+0000 7fe6f51af800 20 remove_watcher() i=0 2023-12-03T11:54:13.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.653+0000 7fe6f51af800 2 removed watcher, disabling cache 2023-12-03T11:54:13.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.654+0000 7fe6f51af800 20 remove_watcher() i=1 2023-12-03T11:54:13.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.655+0000 7fe6f51af800 20 remove_watcher() i=2 2023-12-03T11:54:13.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.656+0000 7fe6f51af800 20 remove_watcher() i=3 2023-12-03T11:54:13.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.657+0000 7fe6f51af800 20 remove_watcher() i=4 2023-12-03T11:54:13.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.658+0000 7fe6f51af800 20 remove_watcher() i=5 2023-12-03T11:54:13.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.659+0000 7fe6f51af800 20 remove_watcher() i=6 2023-12-03T11:54:13.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.660+0000 7fe6f51af800 20 remove_watcher() i=7 2023-12-03T11:54:13.667 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:54:13.694 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:13.694 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.706+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 realm 2023-12-03T11:54:13.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.708+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.709+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.709+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.709+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.711 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.709+0000 7f8e61530800 20 searching for the correct realm 2023-12-03T11:54:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.718+0000 7f8e5d2c6640 0 -- 172.21.15.116:0/976086032 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f8e5006c040 msgr2=0x7f8e5005e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.719+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 zone default found 2023-12-03T11:54:13.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 4 Realm: () 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 zonegroup default 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.720+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.721+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.721+0000 7f8e61530800 20 started sync module instance, tier type = 2023-12-03T11:54:13.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.721+0000 7f8e61530800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.726+0000 7f8e5d2c6640 0 -- 172.21.15.116:0/976086032 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f8e38079e10 msgr2=0x7f8e3807c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=0 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=1 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=2 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=3 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=4 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=5 2023-12-03T11:54:13.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.729+0000 7f8e61530800 20 add_watcher() i=6 2023-12-03T11:54:13.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.730+0000 7f8e61530800 20 add_watcher() i=7 2023-12-03T11:54:13.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.730+0000 7f8e61530800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.732+0000 7f8e22fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.732+0000 7f8e61530800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.732+0000 7f8e61530800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8e61530800 5 note: GC not initialized 2023-12-03T11:54:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8d91ffb640 20 reqs_thread_entry: start 2023-12-03T11:54:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8e61530800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8e61530800 20 Filter name: none 2023-12-03T11:54:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8d837fe640 20 reqs_thread_entry: start 2023-12-03T11:54:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.738+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.739+0000 7f8e61530800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.740+0000 7f8e61530800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.741+0000 7f8e61530800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.741+0000 7f8e22fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.743+0000 7f8e61530800 20 remove_watcher() i=0 2023-12-03T11:54:13.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.743+0000 7f8e61530800 2 removed watcher, disabling cache 2023-12-03T11:54:13.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.744+0000 7f8e61530800 20 remove_watcher() i=1 2023-12-03T11:54:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.745+0000 7f8e61530800 20 remove_watcher() i=2 2023-12-03T11:54:13.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.746+0000 7f8e61530800 20 remove_watcher() i=3 2023-12-03T11:54:13.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.747+0000 7f8e61530800 20 remove_watcher() i=4 2023-12-03T11:54:13.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.748+0000 7f8e61530800 20 remove_watcher() i=5 2023-12-03T11:54:13.750 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.749+0000 7f8e61530800 20 remove_watcher() i=6 2023-12-03T11:54:13.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.750+0000 7f8e61530800 20 remove_watcher() i=7 2023-12-03T11:54:13.755 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.756 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:13.756 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:13.756 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.758 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:13.765 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:13.808 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:54:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.850+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 20 realm 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.851+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.852+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.852+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.853 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.852+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.852+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.853+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.853+0000 7f5aa393a800 20 searching for the correct realm 2023-12-03T11:54:13.863 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.862+0000 7f5a9f606640 0 -- 172.21.15.116:0/2782322767 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f5a9805e070 msgr2=0x7f5a98080400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.863+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 zone default found 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 4 Realm: () 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 zonegroup default 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.864+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.865+0000 7f5aa393a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.865+0000 7f5aa393a800 20 started sync module instance, tier type = 2023-12-03T11:54:13.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.865+0000 7f5aa393a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.870+0000 7f5a9f606640 0 -- 172.21.15.116:0/2782322767 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f5a78079e10 msgr2=0x7f5a7807c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=0 2023-12-03T11:54:13.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=1 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=2 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=3 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=4 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=5 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.874+0000 7f5aa393a800 20 add_watcher() i=6 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.875+0000 7f5aa393a800 20 add_watcher() i=7 2023-12-03T11:54:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.875+0000 7f5aa393a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.876+0000 7f5a657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.877+0000 7f5aa393a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.877+0000 7f5aa393a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.882+0000 7f5aa393a800 5 note: GC not initialized 2023-12-03T11:54:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f59d3fff640 20 reqs_thread_entry: start 2023-12-03T11:54:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 20 Filter name: none 2023-12-03T11:54:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f59d1ffb640 20 reqs_thread_entry: start 2023-12-03T11:54:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:54:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.883+0000 7f5aa393a800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 : miss 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.884+0000 7f5aa393a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 2023-12-03T11:54:13.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5aa393a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5aa393a800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5aa393a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.891 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5aa393a800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18, ofs0, ns] 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5a85ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660524 cookie 94175704251200 notifier 6426 bl.length()=887 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5a85ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5a85ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.890+0000 7f5a85ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.891+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): orig_sources= new_sources= 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.891+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): potential sources added=[] removed=[] 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.891+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): orig_dests= new_dests= 2023-12-03T11:54:13.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.891+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): potential dests added=[] removed=[] 2023-12-03T11:54:13.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.0 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.0 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.1 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.1 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.2 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.2 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.3 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.3 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.4 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.4 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.5 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.5 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.6 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.6 2023-12-03T11:54:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.7 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.7 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.8 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.8 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.9 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.9 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.10 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.10 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.11 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1.11 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.0 2023-12-03T11:54:13.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.893+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.1 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.894+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.2 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.894+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.3 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.895+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.4 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.895+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.5 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.895+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.6 2023-12-03T11:54:13.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.896+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.7 2023-12-03T11:54:13.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.896+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.8 2023-12-03T11:54:13.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.897+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.9 2023-12-03T11:54:13.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.897+0000 7f5aa393a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.18.10 2023-12-03T11:54:13.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.900+0000 7f5aa393a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:13.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.900+0000 7f5aa393a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 2023-12-03T11:54:13.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5aa393a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5aa393a800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5aa393a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5aa393a800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18, ofs0, ns] 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5a857fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660525 cookie 94175704251200 notifier 6426 bl.length()=855 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5a857fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 info.flags=0x17 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5a857fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 to cache LRU end 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.901+0000 7f5a857fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.902+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): orig_sources= new_sources= 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.902+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): potential sources added=[] removed=[] 2023-12-03T11:54:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.902+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): orig_dests= new_dests= 2023-12-03T11:54:13.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.902+0000 7f5aa393a800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.18]): potential dests added=[] removed=[] 2023-12-03T11:54:13.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.904+0000 7f5aa393a800 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2023-12-03T11:54:13.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.905+0000 7f5a657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.907+0000 7f5aa393a800 20 remove_watcher() i=0 2023-12-03T11:54:13.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.907+0000 7f5aa393a800 2 removed watcher, disabling cache 2023-12-03T11:54:13.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.908+0000 7f5aa393a800 20 remove_watcher() i=1 2023-12-03T11:54:13.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.909+0000 7f5aa393a800 20 remove_watcher() i=2 2023-12-03T11:54:13.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.910+0000 7f5aa393a800 20 remove_watcher() i=3 2023-12-03T11:54:13.912 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.911+0000 7f5aa393a800 20 remove_watcher() i=4 2023-12-03T11:54:13.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.912+0000 7f5aa393a800 20 remove_watcher() i=5 2023-12-03T11:54:13.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.913+0000 7f5aa393a800 20 remove_watcher() i=6 2023-12-03T11:54:13.915 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.914+0000 7f5aa393a800 20 remove_watcher() i=7 2023-12-03T11:54:13.921 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:54:13.948 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:13.948 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:13.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.961+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 realm 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.963+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.964+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.964+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.964+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:13.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.964+0000 7f148cf37800 20 searching for the correct realm 2023-12-03T11:54:13.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.973+0000 7f1488ccd640 0 -- 172.21.15.116:0/1088418232 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1484018070 msgr2=0x7f1484058300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.974+0000 7f148cf37800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.974+0000 7f148cf37800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.974+0000 7f148cf37800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.974+0000 7f148cf37800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.974+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.976 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 zone default found 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 4 Realm: () 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 zonegroup default 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.975+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.976+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.976+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.976+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.976+0000 7f148cf37800 20 started sync module instance, tier type = 2023-12-03T11:54:13.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.976+0000 7f148cf37800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:13.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.981+0000 7f1488ccd640 0 -- 172.21.15.116:0/1088418232 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1464079e10 msgr2=0x7f146407c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:13.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=0 2023-12-03T11:54:13.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=1 2023-12-03T11:54:13.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=2 2023-12-03T11:54:13.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=3 2023-12-03T11:54:13.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=4 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=5 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=6 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 20 add_watcher() i=7 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.985+0000 7f148cf37800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.987+0000 7f144e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.987+0000 7f148cf37800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.987+0000 7f148cf37800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.993+0000 7f148cf37800 5 note: GC not initialized 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.993+0000 7f13bd7fa640 20 reqs_thread_entry: start 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.993+0000 7f148cf37800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.993+0000 7f148cf37800 20 Filter name: none 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.993+0000 7f13b6ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.994+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.995+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:13.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.995+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.995+0000 7f148cf37800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:13.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.995+0000 7f148cf37800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:13.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.996+0000 7f148cf37800 20 rados->read ofs=0 len=0 2023-12-03T11:54:13.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.996+0000 7f148cf37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:13.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.997+0000 7f144e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.999+0000 7f148cf37800 20 remove_watcher() i=0 2023-12-03T11:54:14.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:13.999+0000 7f148cf37800 2 removed watcher, disabling cache 2023-12-03T11:54:14.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.000+0000 7f148cf37800 20 remove_watcher() i=1 2023-12-03T11:54:14.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.001+0000 7f148cf37800 20 remove_watcher() i=2 2023-12-03T11:54:14.003 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.002+0000 7f148cf37800 20 remove_watcher() i=3 2023-12-03T11:54:14.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.003+0000 7f148cf37800 20 remove_watcher() i=4 2023-12-03T11:54:14.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.004+0000 7f148cf37800 20 remove_watcher() i=5 2023-12-03T11:54:14.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.005+0000 7f148cf37800 20 remove_watcher() i=6 2023-12-03T11:54:14.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.006+0000 7f148cf37800 20 remove_watcher() i=7 2023-12-03T11:54:14.012 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.012 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:14.012 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.18 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:54:14.012 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.014 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.017 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.062 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.086 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with ECANCELED injected at set_target_layout 2023-12-03T11:54:14.086 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.087 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.097 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.140 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.148 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.155 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.163 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.170 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.178 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.185 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.192 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.201 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.209 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.216 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.224 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.231 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.240 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.247 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.254 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.261 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.268 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.275 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.282 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.290 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:54:14.317 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:14.317 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.329+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 realm 2023-12-03T11:54:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.331+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.332+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.332+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.332+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:14.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.332+0000 7eff37e85800 20 searching for the correct realm 2023-12-03T11:54:14.343 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.341+0000 7eff35985640 0 -- 172.21.15.116:0/3220686221 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x563e4ded2ae0 msgr2=0x563e4ded4ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.343+0000 7eff37e85800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.343+0000 7eff37e85800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.343+0000 7eff37e85800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.343+0000 7eff37e85800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.343+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.344+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.344+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.344+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.344+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 zone default found 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 4 Realm: () 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:14.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 zonegroup default 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 started sync module instance, tier type = 2023-12-03T11:54:14.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.345+0000 7eff37e85800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:14.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.351+0000 7eff36186640 0 -- 172.21.15.116:0/3220686221 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7eff2c039920 msgr2=0x7eff2c026420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=0 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=1 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=2 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=3 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=4 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=5 2023-12-03T11:54:14.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.355+0000 7eff37e85800 20 add_watcher() i=6 2023-12-03T11:54:14.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.356+0000 7eff37e85800 20 add_watcher() i=7 2023-12-03T11:54:14.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.356+0000 7eff37e85800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:14.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.358+0000 7efefb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.358+0000 7eff37e85800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:14.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.358+0000 7eff37e85800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:14.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7eff37e85800 5 note: GC not initialized 2023-12-03T11:54:14.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7efe6a7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:14.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7eff37e85800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:14.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7eff37e85800 20 Filter name: none 2023-12-03T11:54:14.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7efe5bfff640 20 reqs_thread_entry: start 2023-12-03T11:54:14.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.364+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.365+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.365+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.365+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.365+0000 7eff37e85800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.365+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.366+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.366+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.366+0000 7eff37e85800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.366+0000 7eff37e85800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.367+0000 7eff37e85800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.367+0000 7eff37e85800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.368+0000 7efefb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.370+0000 7eff37e85800 20 remove_watcher() i=0 2023-12-03T11:54:14.371 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.370+0000 7eff37e85800 2 removed watcher, disabling cache 2023-12-03T11:54:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.371+0000 7eff37e85800 20 remove_watcher() i=1 2023-12-03T11:54:14.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.372+0000 7eff37e85800 20 remove_watcher() i=2 2023-12-03T11:54:14.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.373+0000 7eff37e85800 20 remove_watcher() i=3 2023-12-03T11:54:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.374+0000 7eff37e85800 20 remove_watcher() i=4 2023-12-03T11:54:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.375+0000 7eff37e85800 20 remove_watcher() i=5 2023-12-03T11:54:14.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.376+0000 7eff37e85800 20 remove_watcher() i=6 2023-12-03T11:54:14.378 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.377+0000 7eff37e85800 20 remove_watcher() i=7 2023-12-03T11:54:14.383 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.384 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:14.384 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:14.384 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.384 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout --inject-error-code 125 2023-12-03T11:54:14.419 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:14.419 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:14.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.433+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 realm 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.434+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.435+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.435+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.435+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:14.437 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.435+0000 7f87827ad800 20 searching for the correct realm 2023-12-03T11:54:14.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.445+0000 7f87802ad640 0 -- 172.21.15.116:0/3437937354 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5635899623e0 msgr2=0x5635899647d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:14.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:14.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.447+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:14.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 zone default found 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 4 Realm: () 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 zonegroup default 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.448+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.449+0000 7f87827ad800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.449+0000 7f87827ad800 20 started sync module instance, tier type = 2023-12-03T11:54:14.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.449+0000 7f87827ad800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.454+0000 7f877faac640 0 -- 172.21.15.116:0/3437937354 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x563589967230 msgr2=0x563589969620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=0 2023-12-03T11:54:14.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=1 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=2 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=3 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=4 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=5 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.457+0000 7f87827ad800 20 add_watcher() i=6 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.458+0000 7f87827ad800 20 add_watcher() i=7 2023-12-03T11:54:14.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.458+0000 7f87827ad800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.460+0000 7f87467e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.460+0000 7f87827ad800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.460+0000 7f87827ad800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f87827ad800 5 note: GC not initialized 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f86b57fa640 20 reqs_thread_entry: start 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f87827ad800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f87827ad800 20 Filter name: none 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f86aeffd640 20 reqs_thread_entry: start 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f87827ad800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:54:14.468 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.466+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 : miss 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 info.flags=0x17 2023-12-03T11:54:14.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 to cache LRU end 2023-12-03T11:54:14.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:14.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:14.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.467+0000 7f87827ad800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 2023-12-03T11:54:14.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.471+0000 7f87827ad800 -1 Injecting error=-125 at location=set_target_layout 2023-12-03T11:54:14.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 info.flags=0x17 2023-12-03T11:54:14.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 to cache LRU end 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19, ofs0, ns] 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87667fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660528 cookie 94787941615824 notifier 6498 bl.length()=887 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87667fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 info.flags=0x17 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87667fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 to cache LRU end 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87667fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): orig_sources= new_sources= 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): potential sources added=[] removed=[] 2023-12-03T11:54:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): orig_dests= new_dests= 2023-12-03T11:54:14.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.473+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): potential dests added=[] removed=[] 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.0 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.0 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.1 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.1 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.2 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.2 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.3 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.3 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.4 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.4 2023-12-03T11:54:14.478 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.5 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.5 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.6 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.6 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.7 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.7 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.8 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.8 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.9 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.9 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.10 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.10 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.11 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1.11 2023-12-03T11:54:14.479 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.476+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.0 2023-12-03T11:54:14.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.477+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.1 2023-12-03T11:54:14.480 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.477+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.2 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.477+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.3 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.478+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.4 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.478+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.5 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.479+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.6 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.479+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.7 2023-12-03T11:54:14.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.480+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.8 2023-12-03T11:54:14.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.480+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.9 2023-12-03T11:54:14.482 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.480+0000 7f87827ad800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.19.10 2023-12-03T11:54:14.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.485+0000 7f87827ad800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:14.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.485+0000 7f87827ad800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 2023-12-03T11:54:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 info.flags=0x17 2023-12-03T11:54:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 to cache LRU end 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19, ofs0, ns] 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f8766ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660529 cookie 94787941615824 notifier 6498 bl.length()=855 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f8766ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 info.flags=0x17 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f8766ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 to cache LRU end 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f8766ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): orig_sources= new_sources= 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): potential sources added=[] removed=[] 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): orig_dests= new_dests= 2023-12-03T11:54:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.487+0000 7f87827ad800 20 handle_bi_update(): bucket=:error-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.19]): potential dests added=[] removed=[] 2023-12-03T11:54:14.491 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.490+0000 7f87827ad800 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2023-12-03T11:54:14.492 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.491+0000 7f87467e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.493+0000 7f87827ad800 20 remove_watcher() i=0 2023-12-03T11:54:14.494 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.493+0000 7f87827ad800 2 removed watcher, disabling cache 2023-12-03T11:54:14.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.494+0000 7f87827ad800 20 remove_watcher() i=1 2023-12-03T11:54:14.496 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.495+0000 7f87827ad800 20 remove_watcher() i=2 2023-12-03T11:54:14.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.496+0000 7f87827ad800 20 remove_watcher() i=3 2023-12-03T11:54:14.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.497+0000 7f87827ad800 20 remove_watcher() i=4 2023-12-03T11:54:14.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.498+0000 7f87827ad800 20 remove_watcher() i=5 2023-12-03T11:54:14.500 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.499+0000 7f87827ad800 20 remove_watcher() i=6 2023-12-03T11:54:14.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.500+0000 7f87827ad800 20 remove_watcher() i=7 2023-12-03T11:54:14.507 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:54:14.534 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:14.534 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:14.548 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.546+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 realm 2023-12-03T11:54:14.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:14.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.548+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.549+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.549+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.549+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:14.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.549+0000 7f331d9af800 20 searching for the correct realm 2023-12-03T11:54:14.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.558+0000 7f331b4af640 0 -- 172.21.15.116:0/2288540210 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x562c0d78c4f0 msgr2=0x562c0d78e8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.560+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 zone default found 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 4 Realm: () 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 zonegroup default 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.561+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.562+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.562+0000 7f331d9af800 20 started sync module instance, tier type = 2023-12-03T11:54:14.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.562+0000 7f331d9af800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:14.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.567+0000 7f331bcb0640 0 -- 172.21.15.116:0/2288540210 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f3310036430 msgr2=0x7f33100368b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=0 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=1 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=2 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=3 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=4 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=5 2023-12-03T11:54:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.570+0000 7f331d9af800 20 add_watcher() i=6 2023-12-03T11:54:14.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.571+0000 7f331d9af800 20 add_watcher() i=7 2023-12-03T11:54:14.573 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.571+0000 7f331d9af800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:14.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.572+0000 7f32e17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.573+0000 7f331d9af800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:14.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.573+0000 7f331d9af800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 5 note: GC not initialized 2023-12-03T11:54:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f324ffff640 20 reqs_thread_entry: start 2023-12-03T11:54:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 20 Filter name: none 2023-12-03T11:54:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f324dffb640 20 reqs_thread_entry: start 2023-12-03T11:54:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.579+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.580+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.580+0000 7f331d9af800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.580+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.580+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.580+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.581+0000 7f331d9af800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.581+0000 7f331d9af800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.581+0000 7f331d9af800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.582+0000 7f331d9af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.582+0000 7f32e17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.584+0000 7f331d9af800 20 remove_watcher() i=0 2023-12-03T11:54:14.586 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.584+0000 7f331d9af800 2 removed watcher, disabling cache 2023-12-03T11:54:14.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.585+0000 7f331d9af800 20 remove_watcher() i=1 2023-12-03T11:54:14.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.586+0000 7f331d9af800 20 remove_watcher() i=2 2023-12-03T11:54:14.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.587+0000 7f331d9af800 20 remove_watcher() i=3 2023-12-03T11:54:14.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.588+0000 7f331d9af800 20 remove_watcher() i=4 2023-12-03T11:54:14.590 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.589+0000 7f331d9af800 20 remove_watcher() i=5 2023-12-03T11:54:14.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.590+0000 7f331d9af800 20 remove_watcher() i=6 2023-12-03T11:54:14.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.591+0000 7f331d9af800 20 remove_watcher() i=7 2023-12-03T11:54:14.597 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.597 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:14.597 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.19 num_objects: 20 size_kb: 1 num_shards: 12 2023-12-03T11:54:14.597 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.599 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.602 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.666 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.691 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with abort at set_target_layout 2023-12-03T11:54:14.691 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.692 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.705 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.748 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.756 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.765 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.773 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.781 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.789 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.796 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.804 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.812 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.819 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.827 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.834 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.842 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.850 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.857 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.865 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.872 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.881 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.890 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:14.897 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:14.905 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:54:14.932 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:14.932 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:14.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.945+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 realm 2023-12-03T11:54:14.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.947+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.948+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.948+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.948+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:14.950 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.948+0000 7f3641b29800 20 searching for the correct realm 2023-12-03T11:54:14.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.955+0000 7f363e0c0640 0 -- 172.21.15.116:0/4283001420 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f3634033930 msgr2=0x7f3634050400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.958+0000 7f3641b29800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.958+0000 7f3641b29800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.958+0000 7f3641b29800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.958+0000 7f3641b29800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.958+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:14.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f363e0c0640 0 -- 172.21.15.116:0/4283001420 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f3634042940 msgr2=0x7f3634017430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 zone default found 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 4 Realm: () 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 zonegroup default 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.959+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.960+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.960+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.960+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.960+0000 7f3641b29800 20 started sync module instance, tier type = 2023-12-03T11:54:14.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.960+0000 7f3641b29800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:14.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=0 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=1 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=2 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=3 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=4 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=5 2023-12-03T11:54:14.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.969+0000 7f3641b29800 20 add_watcher() i=6 2023-12-03T11:54:14.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.971+0000 7f3641b29800 20 add_watcher() i=7 2023-12-03T11:54:14.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.971+0000 7f3641b29800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:14.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.972+0000 7f36037e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.973+0000 7f3641b29800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:14.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.973+0000 7f3641b29800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.978+0000 7f3641b29800 5 note: GC not initialized 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.978+0000 7f35727fc640 20 reqs_thread_entry: start 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.978+0000 7f3641b29800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.979+0000 7f3641b29800 20 Filter name: none 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.979+0000 7f3563fff640 20 reqs_thread_entry: start 2023-12-03T11:54:14.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.979+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.979+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.979+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.980+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.980+0000 7f3641b29800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.980+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.980+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.980+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.981+0000 7f3641b29800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:14.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.981+0000 7f3641b29800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:14.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.982+0000 7f3641b29800 20 rados->read ofs=0 len=0 2023-12-03T11:54:14.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.982+0000 7f3641b29800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:14.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.983+0000 7f36037e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:14.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.984+0000 7f3641b29800 20 remove_watcher() i=0 2023-12-03T11:54:14.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.984+0000 7f3641b29800 2 removed watcher, disabling cache 2023-12-03T11:54:14.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.986+0000 7f3641b29800 20 remove_watcher() i=1 2023-12-03T11:54:14.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.987+0000 7f3641b29800 20 remove_watcher() i=2 2023-12-03T11:54:14.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.987+0000 7f3641b29800 20 remove_watcher() i=3 2023-12-03T11:54:14.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.988+0000 7f3641b29800 20 remove_watcher() i=4 2023-12-03T11:54:14.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.989+0000 7f3641b29800 20 remove_watcher() i=5 2023-12-03T11:54:14.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.990+0000 7f3641b29800 20 remove_watcher() i=6 2023-12-03T11:54:14.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:14.991+0000 7f3641b29800 20 remove_watcher() i=7 2023-12-03T11:54:14.997 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.997 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:14.997 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:14.997 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:14.998 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at set_target_layout 2023-12-03T11:54:15.030 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:15.030 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:15.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.043+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 20 realm 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.046+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.046+0000 7f9f86524800 20 searching for the correct realm 2023-12-03T11:54:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.053+0000 7f9f821c9640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f9f7800d3b0 msgr2=0x7f9f7800d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:15.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 zone default found 2023-12-03T11:54:15.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 4 Realm: () 2023-12-03T11:54:15.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 zonegroup default 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 started sync module instance, tier type = 2023-12-03T11:54:15.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.057+0000 7f9f86524800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:15.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.058+0000 7f9f829ca640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9f7802e950 msgr2=0x7f9f7802d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=0 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=1 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=2 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=3 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=4 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=5 2023-12-03T11:54:15.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=6 2023-12-03T11:54:15.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=7 2023-12-03T11:54:15.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.067+0000 7f9f86524800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:15.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.068+0000 7f9f47fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:15.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9f86524800 5 note: GC not initialized 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9eb6ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9f86524800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9f86524800 20 Filter name: none 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9eb4ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:54:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.075+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : miss 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:15.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.076+0000 7f9f86524800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:15.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f9f86524800 time 2023-12-03T11:54:15.081484+0000 2023-12-03T11:54:15.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f9f8916eef8] 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 4: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.082 INFO:tasks.workunit.client.0.smithi116.stderr: 6: main() 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 8: __libc_start_main() 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 9: _start() 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f9f86524800 thread_name:radosgw-admin 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.081+0000 7f9f86524800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f9f86524800 time 2023-12-03T11:54:15.081484+0000 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f9f8916eef8] 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.083 INFO:tasks.workunit.client.0.smithi116.stderr: 4: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 6: main() 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 8: __libc_start_main() 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 9: _start() 2023-12-03T11:54:15.084 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9f86a54db0] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9f86aa154c] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9f8916ef52] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 8: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 10: main() 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 12: __libc_start_main() 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 13: _start() 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.082+0000 7f9f86524800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f9f86524800 thread_name:radosgw-admin 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9f86a54db0] 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9f86aa154c] 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9f8916ef52] 2023-12-03T11:54:15.104 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 8: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 10: main() 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 12: __libc_start_main() 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 13: _start() 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.105 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6555 payload 824 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:54:15.037+0000 7f9f821c9640 1 monclient: found mon.noname-a 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: dump: 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.106 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 monclient: authenticate success, global_id 6555 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: dump: 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.107 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:54:15.038+0000 7f9f809c6640 4 set_mon_vals no callback set 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:54:15.038+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: get_monmap_and_config 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:54:15.038+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.108 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.109 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.110 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.111 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x7ffca74d2db8 EEXIST 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f8611b20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f861d3c0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8611f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.345384s 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8611b20 auth_method 0 2023-12-03T11:54:15.112 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f8611f00 auth_method 0 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.113 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_done global_id 6203 payload 274 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:54:15.039+0000 7f9f829ca640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:54:15.039+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f861d3c0 auth_method 0 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:54:15.040+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f64069820 auth_method 0 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: dump: 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: dump: 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 monclient: authenticate success, global_id 6203 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:54:15.040+0000 7f9f86524800 4 set_mon_vals no callback set 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: shutdown 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:54:15.040+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.117 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: starting msgr at 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: starting objecter 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command objecter_requests cmddesc objecter_requests hook 0x55d8f860e0b0 EEXIST 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: setting wanted keys 2023-12-03T11:54:15.118 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: calling monclient init 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.119 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x55d8f85da8a0 EEXIST 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8611f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f861d3c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f8609f90 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.120 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.347465s 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f8611f00 auth_method 0 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f861d3c0 auth_method 0 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8609f90 auth_method 0 2023-12-03T11:54:15.121 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6561 payload 824 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:54:15.042+0000 7f9f821c9640 1 monclient: found mon.noname-a 2023-12-03T11:54:15.122 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: dump: 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.123 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: dump: 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:54:15.042+0000 7f9f86524800 5 monclient: authenticate success, global_id 6561 2023-12-03T11:54:15.124 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:54:15.042+0000 7f9f70ff9640 4 set_mon_vals no callback set 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:54:15.042+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:54:15.043+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:54:15.044+0000 7f9f819c8640 10 monclient: get_auth_request con 0x7f9f5c069820 auth_method 0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:54:15.044+0000 7f9f829ca640 10 monclient: get_auth_request con 0x55d8f862f7f0 auth_method 0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 realm 2023-12-03T11:54:15.125 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:54:15.044+0000 7f9f86524800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:54:15.045+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f8633600 auth_method 0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:54:15.046+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:54:15.046+0000 7f9f86524800 20 searching for the correct realm 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:54:15.046+0000 7f9f819c8640 10 monclient: get_auth_request con 0x55d8f8638640 auth_method 0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:54:15.047+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f6c0030b0 auth_method 0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:54:15.047+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f6c003b70 auth_method 0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:54:15.048+0000 7f9f819c8640 10 monclient: get_auth_request con 0x7f9f740d1040 auth_method 0 2023-12-03T11:54:15.126 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:54:15.049+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f7802e950 auth_method 0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:54:15.051+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f7800d3b0 auth_method 0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:54:15.053+0000 7f9f821c9640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f9f7800d3b0 msgr2=0x7f9f7800d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 1 client.6561.objecter ms_handle_reset 0x7f9f7800d3b0 session 0x7f9f7802edd0 osd.0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 10 monclient: _renew_subs 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:54:15.053+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f5c079d90 auth_method 0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.127 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 zone default found 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 Realm: () 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:54:15.057+0000 7f9f86524800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 zonegroup default 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 started sync module instance, tier type = 2023-12-03T11:54:15.128 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:54:15.058+0000 7f9f829ca640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9f7802e950 msgr2=0x7f9f7802d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:54:15.058+0000 7f9f62ffd640 1 client.6561.objecter ms_handle_reset 0x7f9f7802e950 session 0x7f9f7801bde0 osd.6 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:54:15.059+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f7800d3b0 auth_method 0 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=0 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=1 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=2 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=3 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=4 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=5 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=6 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=7 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:54:15.067+0000 7f9f86524800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.129 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:54:15.068+0000 7f9f47fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.130 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:54:15.068+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache list hook 0x55d8f85d7130 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache inspect hook 0x55d8f85d7130 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache erase hook 0x55d8f85d7130 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache zap hook 0x55d8f85d7130 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: get_monmap_and_config 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:54:15.068+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.131 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.132 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.133 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x7ffca74d3338 EEXIST 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.134 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8677ba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f8677180 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f86790d0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.375464s 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f8677180 auth_method 0 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8677ba0 auth_method 0 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.135 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:54:15.070+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f86790d0 auth_method 0 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6209 payload 274 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:54:15.070+0000 7f9f821c9640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.136 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: dump: 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.137 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: dump: 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:54:15.070+0000 7f9f86524800 5 monclient: authenticate success, global_id 6209 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:54:15.070+0000 7f9f86524800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:15.138 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:54:15.070+0000 7f9f86524800 4 set_mon_vals no callback set 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: shutdown 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.139 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.140 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:54:15.071+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: starting msgr at 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: starting objecter 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command objecter_requests cmddesc objecter_requests hook 0x55d8f866f650 EEXIST 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: setting wanted keys 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: calling monclient init 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.141 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:54:15.072+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:54:15.072+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:54:15.072+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x55d8f8679cb0 EEXIST 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8677ba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.142 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f86790d0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f866b7a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.378345s 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f866b7a0 auth_method 0 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f86790d0 auth_method 0 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.143 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8677ba0 auth_method 0 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.144 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient(hunting): handle_auth_done global_id 6212 payload 824 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:54:15.073+0000 7f9f819c8640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: dump: 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.145 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: dump: 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.146 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:54:15.073+0000 7f9f86524800 5 monclient: authenticate success, global_id 6212 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:54:15.073+0000 7f9f477e6640 4 set_mon_vals no callback set 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:54:15.073+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:54:15.073+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cr dump hook 0x55d8f866a338 2023-12-03T11:54:15.147 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:54:15.075+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9ebc069820 auth_method 0 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 note: GC not initialized 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:54:15.075+0000 7f9eb6ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace show hook 0x7f9f7400b440 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace history hook 0x7f9f7400b440 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace active hook 0x7f9f7400b440 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace active_short hook 0x7f9f7400b440 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 Filter name: none 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:54:15.075+0000 7f9eb4ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:54:15.075+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.148 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : miss 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:54:15.076+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f86ca9f0 auth_method 0 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:54:15.081+0000 7f9f86524800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f9f86524800 time 2023-12-03T11:54:15.081484+0000 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:15.149 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f9f8916eef8] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 4: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 6: main() 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 8: __libc_start_main() 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 9: _start() 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:54:15.082+0000 7f9f86524800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f9f86524800 thread_name:radosgw-admin 2023-12-03T11:54:15.150 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9f86a54db0] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9f86aa154c] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9f8916ef52] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 8: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 10: main() 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 12: __libc_start_main() 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 13: _start() 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:54:15.152 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:54:15.153 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:54:15.154 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:54:15.155 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:54:15.156 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:54:15.157 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9eb4ff9640 / http_manager 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9eb6ffd640 / http_manager 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f45fe3640 / ms_dispatch 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f477e6640 / io_context_pool 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f47fe7640 / rgw_dt_lg_renew 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f62ffd640 / ms_dispatch 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f70ff9640 / io_context_pool 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f737fe640 / ms_dispatch 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f809c6640 / io_context_pool 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f819c8640 / msgr-worker-2 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f821c9640 / msgr-worker-1 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f829ca640 / msgr-worker-0 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f86524800 / radosgw-admin 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:54:15.158 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/log/ceph/ceph-client.admin.101643.log 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:54:15.037+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6555 payload 824 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:54:15.037+0000 7f9f821c9640 1 monclient: found mon.noname-a 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:54:15.037+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: dump: 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 monclient: authenticate success, global_id 6555 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.160 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:54:15.038+0000 7f9f737fe640 10 monclient: dump: 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:54:15.038+0000 7f9f809c6640 4 set_mon_vals no callback set 2023-12-03T11:54:15.161 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:54:15.038+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: get_monmap_and_config 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:54:15.038+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:54:15.038+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.162 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding auth protocol: cephx 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.163 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:54:15.038+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e2ff0) adding con mode: secure 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding auth protocol: cephx 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.164 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: crc 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d2c70) adding con mode: secure 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:54:15.039+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:54:15.039+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x7ffca74d2db8 EEXIST 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f8611b20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f861d3c0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8611f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.165 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:54:15.039+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.345384s 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8611b20 auth_method 0 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:54:15.039+0000 7f9f737fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f8611f00 auth_method 0 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.166 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:54:15.039+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient(hunting): handle_auth_done global_id 6203 payload 274 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:54:15.039+0000 7f9f829ca640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:54:15.039+0000 7f9f829ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:54:15.039+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f861d3c0 auth_method 0 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:54:15.040+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f64069820 auth_method 0 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: dump: 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.167 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:54:15.040+0000 7f9f70ff9640 10 monclient: dump: 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.168 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 monclient: authenticate success, global_id 6203 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:54:15.040+0000 7f9f86524800 4 set_mon_vals no callback set 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: shutdown 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:54:15.040+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:54:15.040+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.169 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding auth protocol: cephx 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.170 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: crc 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:54:15.040+0000 7f9f86524800 5 AuthRegistry(0x55d8f85e4e60) adding con mode: secure 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: starting msgr at 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: starting objecter 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command objecter_requests cmddesc objecter_requests hook 0x55d8f860e0b0 EEXIST 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: setting wanted keys 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:54:15.041+0000 7f9f86524800 1 librados: calling monclient init 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.171 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding auth protocol: cephx 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: crc 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 AuthRegistry(0x55d8f85da758) adding con mode: secure 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:54:15.041+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.172 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:54:15.041+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x55d8f85da8a0 EEXIST 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8611f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f861d3c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f8609f90 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:54:15.041+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.347465s 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f8611f00 auth_method 0 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f861d3c0 auth_method 0 2023-12-03T11:54:15.173 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:54:15.041+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:54:15.041+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8609f90 auth_method 0 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:54:15.041+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.174 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:54:15.042+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6561 payload 824 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:54:15.042+0000 7f9f821c9640 1 monclient: found mon.noname-a 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:54:15.042+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: dump: 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.175 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:54:15.042+0000 7f9f62ffd640 10 monclient: dump: 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.176 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:54:15.042+0000 7f9f86524800 5 monclient: authenticate success, global_id 6561 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:54:15.042+0000 7f9f70ff9640 4 set_mon_vals no callback set 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:54:15.042+0000 7f9f86524800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:54:15.042+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:54:15.043+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:54:15.043+0000 7f9f62ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:54:15.044+0000 7f9f819c8640 10 monclient: get_auth_request con 0x7f9f5c069820 auth_method 0 2023-12-03T11:54:15.177 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:54:15.044+0000 7f9f829ca640 10 monclient: get_auth_request con 0x55d8f862f7f0 auth_method 0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 realm 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:54:15.044+0000 7f9f86524800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:54:15.044+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:54:15.045+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f8633600 auth_method 0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:54:15.045+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:54:15.046+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:15.178 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:54:15.046+0000 7f9f86524800 20 searching for the correct realm 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:54:15.046+0000 7f9f819c8640 10 monclient: get_auth_request con 0x55d8f8638640 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:54:15.047+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f6c0030b0 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:54:15.047+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f6c003b70 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:54:15.048+0000 7f9f819c8640 10 monclient: get_auth_request con 0x7f9f740d1040 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:54:15.049+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f7802e950 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:54:15.051+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f7800d3b0 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:54:15.053+0000 7f9f821c9640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f9f7800d3b0 msgr2=0x7f9f7800d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 1 client.6561.objecter ms_handle_reset 0x7f9f7800d3b0 session 0x7f9f7802edd0 osd.0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 10 monclient: _renew_subs 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:54:15.053+0000 7f9f62ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:54:15.053+0000 7f9f821c9640 10 monclient: get_auth_request con 0x7f9f5c079d90 auth_method 0 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:15.179 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:54:15.056+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 zone default found 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 Realm: () 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:54:15.057+0000 7f9f86524800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:54:15.057+0000 7f9f86524800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 zonegroup default 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.180 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 started sync module instance, tier type = 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:54:15.057+0000 7f9f86524800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:54:15.058+0000 7f9f829ca640 0 -- 172.21.15.116:0/3042703420 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9f7802e950 msgr2=0x7f9f7802d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:54:15.058+0000 7f9f62ffd640 1 client.6561.objecter ms_handle_reset 0x7f9f7802e950 session 0x7f9f7801bde0 osd.6 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:54:15.059+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9f7800d3b0 auth_method 0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=0 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=1 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=2 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=3 2023-12-03T11:54:15.181 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=4 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=5 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=6 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:54:15.066+0000 7f9f86524800 20 add_watcher() i=7 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:54:15.067+0000 7f9f86524800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding auth protocol: cephx 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:54:15.068+0000 7f9f47fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.182 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: crc 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d36a0) adding con mode: secure 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:54:15.068+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:54:15.068+0000 7f9f86524800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache list hook 0x55d8f85d7130 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache inspect hook 0x55d8f85d7130 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache erase hook 0x55d8f85d7130 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cache zap hook 0x55d8f85d7130 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: get_monmap_and_config 2023-12-03T11:54:15.183 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:54:15.068+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:54:15.068+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.184 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding auth protocol: cephx 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: crc 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:54:15.068+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676650) adding con mode: secure 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.185 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding auth protocol: cephx 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: crc 2023-12-03T11:54:15.186 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 AuthRegistry(0x7ffca74d31f0) adding con mode: secure 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:54:15.069+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:54:15.069+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x7ffca74d3338 EEXIST 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8677ba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f8677180 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f86790d0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:54:15.069+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.375464s 2023-12-03T11:54:15.187 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f8677180 auth_method 0 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8677ba0 auth_method 0 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:54:15.069+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:54:15.069+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:54:15.070+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f86790d0 auth_method 0 2023-12-03T11:54:15.188 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:54:15.070+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient(hunting): handle_auth_done global_id 6209 payload 274 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:54:15.070+0000 7f9f821c9640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:54:15.070+0000 7f9f821c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: dump: 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:54:15.070+0000 7f9f477e6640 10 monclient: dump: 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:54:15.070+0000 7f9f86524800 5 monclient: authenticate success, global_id 6209 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:54:15.070+0000 7f9f86524800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:54:15.070+0000 7f9f86524800 4 set_mon_vals no callback set 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: shutdown 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: build_initial_monmap 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 build_initial for_mkfs: 0 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: monmap: 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding auth protocol: cephx 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.192 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: crc 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8676560) adding con mode: secure 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:54:15.071+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: starting msgr at 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: starting objecter 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command objecter_requests cmddesc objecter_requests hook 0x55d8f866f650 EEXIST 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: setting wanted keys 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:54:15.071+0000 7f9f86524800 1 librados: calling monclient init 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:54:15.071+0000 7f9f86524800 10 monclient: init 2023-12-03T11:54:15.193 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding auth protocol: cephx 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: crc 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:54:15.071+0000 7f9f86524800 5 AuthRegistry(0x55d8f8679b68) adding con mode: secure 2023-12-03T11:54:15.194 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:54:15.072+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:54:15.072+0000 7f9f86524800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:54:15.072+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command rotate-key cmddesc rotate-key hook 0x55d8f8679cb0 EEXIST 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-a con 0x55d8f8677ba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-b con 0x55d8f86790d0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): picked mon.noname-c con 0x55d8f866b7a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:54:15.072+0000 7f9f86524800 10 monclient(hunting): authenticate will time out at 1511.378345s 2023-12-03T11:54:15.195 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): get_auth_request con 0x55d8f866b7a0 auth_method 0 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): get_auth_request con 0x55d8f86790d0 auth_method 0 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:54:15.072+0000 7f9f821c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): get_auth_request con 0x55d8f8677ba0 auth_method 0 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:15.196 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:54:15.072+0000 7f9f829ca640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:54:15.072+0000 7f9f819c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:54:15.073+0000 7f9f821c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:54:15.073+0000 7f9f829ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient(hunting): handle_auth_done global_id 6212 payload 824 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient: _finish_hunting 0 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:54:15.073+0000 7f9f819c8640 1 monclient: found mon.noname-c 2023-12-03T11:54:15.197 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:54:15.073+0000 7f9f819c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: dump: 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: _finish_auth 0 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: _check_auth_tickets 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:54:15.073+0000 7f9f45fe3640 10 monclient: dump: 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:54:15.073+0000 7f9f86524800 5 monclient: authenticate success, global_id 6212 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:54:15.073+0000 7f9f477e6640 4 set_mon_vals no callback set 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _renew_subs 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:54:15.073+0000 7f9f86524800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:54:15.073+0000 7f9f86524800 1 librados: init done 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:54:15.073+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command cr dump hook 0x55d8f866a338 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:54:15.075+0000 7f9f45fe3640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:15.200 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:54:15.075+0000 7f9f829ca640 10 monclient: get_auth_request con 0x7f9ebc069820 auth_method 0 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 note: GC not initialized 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:54:15.075+0000 7f9eb6ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace show hook 0x7f9f7400b440 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace history hook 0x7f9f7400b440 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace active hook 0x7f9f7400b440 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:54:15.075+0000 7f9f86524800 5 asok(0x55d8f83d8a10) register_command sync trace active_short hook 0x7f9f7400b440 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 Filter name: none 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:54:15.075+0000 7f9eb4ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:54:15.075+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:54:15.075+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:54:15.201 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : miss 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:54:15.076+0000 7f9f86524800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:54:15.076+0000 7f9f86524800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:54:15.076+0000 7f9f821c9640 10 monclient: get_auth_request con 0x55d8f86ca9f0 auth_method 0 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:54:15.081+0000 7f9f86524800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f9f86524800 time 2023-12-03T11:54:15.081484+0000 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.202 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f9f8916eef8] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 4: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 6: main() 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 8: __libc_start_main() 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 9: _start() 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:54:15.082+0000 7f9f86524800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f9f86524800 thread_name:radosgw-admin 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9f86a54db0] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9f86aa154c] 2023-12-03T11:54:15.203 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9f8916ef52] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f9f8916f0b6] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8f7733a71] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 8: radosgw-admin(+0x79787a) [0x55d8f773b87a] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8f773fe6b] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 10: main() 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f9f86a3feb0] 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 12: __libc_start_main() 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 13: _start() 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:54:15.204 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:54:15.205 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:54:15.206 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:54:15.207 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:54:15.208 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:54:15.209 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9eb4ff9640 / http_manager 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9eb6ffd640 / http_manager 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f45fe3640 / ms_dispatch 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f477e6640 / io_context_pool 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f47fe7640 / rgw_dt_lg_renew 2023-12-03T11:54:15.210 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f62ffd640 / ms_dispatch 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f70ff9640 / io_context_pool 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f737fe640 / ms_dispatch 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f809c6640 / io_context_pool 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f819c8640 / msgr-worker-2 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f821c9640 / msgr-worker-1 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f829ca640 / msgr-worker-0 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: 7f9f86524800 / radosgw-admin 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:54:15.083029Z_a2106d60-258b-49ee-b679-1711c91ad58e/log 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:54:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.127+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.129+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.129+0000 7f80eb52f800 20 realm 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.129+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.130+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.130+0000 7f80eb52f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.130+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.130+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.130+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.131+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.131+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.131+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.131+0000 7f80eb52f800 20 searching for the correct realm 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.140+0000 7f80e7268640 0 -- 172.21.15.116:0/2221340201 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f80e005e040 msgr2=0x7f80e0069460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.142+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 zone default found 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 4 Realm: () 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 zonegroup default 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.143+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.144+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.144+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.144+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.144+0000 7f80eb52f800 20 started sync module instance, tier type = 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.144+0000 7f80eb52f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.149+0000 7f80e7a69640 0 -- 172.21.15.116:0/2221340201 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f80d0058320 msgr2=0x7f80d0070950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=0 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=1 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=2 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=3 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=4 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=5 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.152+0000 7f80eb52f800 20 add_watcher() i=6 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.153+0000 7f80eb52f800 20 add_watcher() i=7 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.153+0000 7f80eb52f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.154+0000 7f80ad7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.155+0000 7f80eb52f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.155+0000 7f80eb52f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.160+0000 7f80eb52f800 5 note: GC not initialized 2023-12-03T11:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f801bfff640 20 reqs_thread_entry: start 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f80eb52f800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f80eb52f800 20 Filter name: none 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f8019ffb640 20 reqs_thread_entry: start 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.161+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.162+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.162+0000 7f80eb52f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.162+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.162+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.162+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.163+0000 7f80eb52f800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.163+0000 7f80eb52f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.163+0000 7f80eb52f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.163+0000 7f80eb52f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.164+0000 7f80ad7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.166+0000 7f80eb52f800 20 remove_watcher() i=0 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.166+0000 7f80eb52f800 2 removed watcher, disabling cache 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.167+0000 7f80eb52f800 20 remove_watcher() i=1 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.168+0000 7f80eb52f800 20 remove_watcher() i=2 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.169+0000 7f80eb52f800 20 remove_watcher() i=3 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.170+0000 7f80eb52f800 20 remove_watcher() i=4 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.171+0000 7f80eb52f800 20 remove_watcher() i=5 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.172+0000 7f80eb52f800 20 remove_watcher() i=6 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.173+0000 7f80eb52f800 20 remove_watcher() i=7 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:15.222 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:15.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:15.223 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:15.231 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:54:15.260 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:15.261 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:15.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.273+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.274+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.274+0000 7fb7dbf22800 20 realm 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.274+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.275+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.275+0000 7fb7dbf22800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:15.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.275+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.275+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.275+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.276+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.276+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.276+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:15.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.276+0000 7fb7dbf22800 20 searching for the correct realm 2023-12-03T11:54:15.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.285+0000 7fb7d73a5640 0 -- 172.21.15.116:0/2905931880 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x557f3ed057f0 msgr2=0x557f3ed07be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:15.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:15.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:15.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:15.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:15.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.286+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:15.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 zone default found 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 4 Realm: () 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 zonegroup default 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 started sync module instance, tier type = 2023-12-03T11:54:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.287+0000 7fb7dbf22800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:15.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.295+0000 7fb7d73a5640 0 -- 172.21.15.116:0/2905931880 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7fb7b0079d90 msgr2=0x7fb7b007c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=0 2023-12-03T11:54:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=1 2023-12-03T11:54:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=2 2023-12-03T11:54:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=3 2023-12-03T11:54:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=4 2023-12-03T11:54:15.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=5 2023-12-03T11:54:15.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.296+0000 7fb7dbf22800 20 add_watcher() i=6 2023-12-03T11:54:15.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.297+0000 7fb7dbf22800 20 add_watcher() i=7 2023-12-03T11:54:15.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.297+0000 7fb7dbf22800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:15.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.299+0000 7fb79dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.299+0000 7fb7dbf22800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:15.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.299+0000 7fb7dbf22800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb7dbf22800 5 note: GC not initialized 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb714ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb7dbf22800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb7dbf22800 20 Filter name: none 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb70a7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb7dbf22800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:54:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.305+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : miss 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 20 rados->read ofs=0 len=0 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.306+0000 7fb7dbf22800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:15.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.308+0000 7fb7dbf22800 0 INFO: RGWReshardLock::lock found lock on abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to be held by another RGW process; skipping for now 2023-12-03T11:54:15.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.308+0000 7fb79dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:15.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.310+0000 7fb7dbf22800 20 remove_watcher() i=0 2023-12-03T11:54:15.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.310+0000 7fb7dbf22800 2 removed watcher, disabling cache 2023-12-03T11:54:15.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.311+0000 7fb7dbf22800 20 remove_watcher() i=1 2023-12-03T11:54:15.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.312+0000 7fb7dbf22800 20 remove_watcher() i=2 2023-12-03T11:54:15.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.313+0000 7fb7dbf22800 20 remove_watcher() i=3 2023-12-03T11:54:15.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.314+0000 7fb7dbf22800 20 remove_watcher() i=4 2023-12-03T11:54:15.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.315+0000 7fb7dbf22800 20 remove_watcher() i=5 2023-12-03T11:54:15.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.316+0000 7fb7dbf22800 20 remove_watcher() i=6 2023-12-03T11:54:15.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:15.317+0000 7fb7dbf22800 20 remove_watcher() i=7 2023-12-03T11:54:15.323 INFO:tasks.workunit.client.0.smithi116.stderr:waiting 30 seconds for reshard lock to expire... 2023-12-03T11:54:45.352 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:54:45.386 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:45.386 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:45.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.401+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 realm 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.402+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.404+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.404+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.405+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.405+0000 7f729ec10800 20 searching for the correct realm 2023-12-03T11:54:45.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.416+0000 7f729a250640 0 -- 172.21.15.116:0/871733378 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5608b26d28b0 msgr2=0x5608b26d4ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.418+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 zone default found 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 4 Realm: () 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 zonegroup default 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.419+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.420+0000 7f729ec10800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.420+0000 7f729ec10800 20 started sync module instance, tier type = 2023-12-03T11:54:45.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.420+0000 7f729ec10800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:45.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.426+0000 7f729aa51640 0 -- 172.21.15.116:0/871733378 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f7290063040 msgr2=0x7f72900b18b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=0 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=1 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=2 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=3 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=4 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=5 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.430+0000 7f729ec10800 20 add_watcher() i=6 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.431+0000 7f729ec10800 20 add_watcher() i=7 2023-12-03T11:54:45.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.431+0000 7f729ec10800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:45.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.433+0000 7f725ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.433 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.433+0000 7f729ec10800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:45.434 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.433+0000 7f729ec10800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.439+0000 7f729ec10800 5 note: GC not initialized 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.439+0000 7f71ceffd640 20 reqs_thread_entry: start 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.439+0000 7f729ec10800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 20 Filter name: none 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f71ccff9640 20 reqs_thread_entry: start 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:54:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:54:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:54:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : miss 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.440+0000 7f729ec10800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:45.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.441+0000 7f729ec10800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:45.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f729ec10800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f729ec10800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f729ec10800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f729ec10800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20, ofs0, ns] 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f727bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660555 cookie 94595353258960 notifier 6624 bl.length()=887 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f727bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.446+0000 7f727bfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.447+0000 7f727bfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.447+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): orig_sources= new_sources= 2023-12-03T11:54:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.447+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): potential sources added=[] removed=[] 2023-12-03T11:54:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.447+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): orig_dests= new_dests= 2023-12-03T11:54:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.447+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): potential dests added=[] removed=[] 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.0 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.0 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.1 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.1 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.2 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.2 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.3 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.3 2023-12-03T11:54:45.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.4 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.4 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.5 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.5 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.6 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.6 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.7 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.7 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.8 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.8 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.9 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.9 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.10 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.10 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.11 2023-12-03T11:54:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1.11 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.449+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.0 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.450+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.1 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.450+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.2 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.451+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.3 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.451+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.4 2023-12-03T11:54:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.452+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.5 2023-12-03T11:54:45.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.452+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.6 2023-12-03T11:54:45.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.453+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.7 2023-12-03T11:54:45.453 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.453+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.8 2023-12-03T11:54:45.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.454+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.9 2023-12-03T11:54:45.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.455+0000 7f729ec10800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.20.10 2023-12-03T11:54:45.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.457+0000 7f729ec10800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:45.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.457+0000 7f729ec10800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 2023-12-03T11:54:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.458+0000 7f729ec10800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20, ofs0, ns] 2023-12-03T11:54:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f727bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660556 cookie 94595353258960 notifier 6624 bl.length()=855 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f727bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 info.flags=0x17 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f727bfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 to cache LRU end 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f727bfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): orig_sources= new_sources= 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): potential sources added=[] removed=[] 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): orig_dests= new_dests= 2023-12-03T11:54:45.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.459+0000 7f729ec10800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.20]): potential dests added=[] removed=[] 2023-12-03T11:54:45.462 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.461+0000 7f729ec10800 1 execute INFO: reshard of bucket "abort-at-set-target-layout" completed successfully 2023-12-03T11:54:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.463+0000 7f725ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.465+0000 7f729ec10800 20 remove_watcher() i=0 2023-12-03T11:54:45.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.465+0000 7f729ec10800 2 removed watcher, disabling cache 2023-12-03T11:54:45.467 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.467+0000 7f729ec10800 20 remove_watcher() i=1 2023-12-03T11:54:45.469 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.469+0000 7f729ec10800 20 remove_watcher() i=2 2023-12-03T11:54:45.470 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.470+0000 7f729ec10800 20 remove_watcher() i=3 2023-12-03T11:54:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.471+0000 7f729ec10800 20 remove_watcher() i=4 2023-12-03T11:54:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.472+0000 7f729ec10800 20 remove_watcher() i=5 2023-12-03T11:54:45.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.472+0000 7f729ec10800 20 remove_watcher() i=6 2023-12-03T11:54:45.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.473+0000 7f729ec10800 20 remove_watcher() i=7 2023-12-03T11:54:45.480 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:54:45.507 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:45.507 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:45.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.520+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 20 realm 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.522+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.523+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.523 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.523+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.523+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.523+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.524+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:45.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.524+0000 7f0fb1925800 20 searching for the correct realm 2023-12-03T11:54:45.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fad54d640 0 -- 172.21.15.116:0/240378938 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5564e3fd28e0 msgr2=0x5564e3fd4cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:45.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:45.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.534+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 zone default found 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 4 Realm: () 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 zonegroup default 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.535+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.536+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.536+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.536+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.536+0000 7f0fb1925800 20 started sync module instance, tier type = 2023-12-03T11:54:45.538 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.536+0000 7f0fb1925800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:45.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.543+0000 7f0fa7fff640 0 -- 172.21.15.116:0/240378938 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5564e3fd7870 msgr2=0x5564e3fd9c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=0 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=1 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=2 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=3 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=4 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.544+0000 7f0fb1925800 20 add_watcher() i=5 2023-12-03T11:54:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.545+0000 7f0fb1925800 20 add_watcher() i=6 2023-12-03T11:54:45.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.545+0000 7f0fb1925800 20 add_watcher() i=7 2023-12-03T11:54:45.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.545+0000 7f0fb1925800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:45.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.547+0000 7f0f72fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.547+0000 7f0fb1925800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:45.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.547+0000 7f0fb1925800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0fb1925800 5 note: GC not initialized 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0ee1ffb640 20 reqs_thread_entry: start 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0fb1925800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0fb1925800 20 Filter name: none 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0ed37fe640 20 reqs_thread_entry: start 2023-12-03T11:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.553+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.554+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.554+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.554+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.554+0000 7f0fb1925800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.554+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.555+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:54:45.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.555+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.555+0000 7f0fb1925800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:54:45.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.555+0000 7f0fb1925800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.556+0000 7f0fb1925800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.557+0000 7f0fb1925800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.557+0000 7f0f72fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.559+0000 7f0fb1925800 20 remove_watcher() i=0 2023-12-03T11:54:45.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.559+0000 7f0fb1925800 2 removed watcher, disabling cache 2023-12-03T11:54:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.560+0000 7f0fb1925800 20 remove_watcher() i=1 2023-12-03T11:54:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.561+0000 7f0fb1925800 20 remove_watcher() i=2 2023-12-03T11:54:45.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.562+0000 7f0fb1925800 20 remove_watcher() i=3 2023-12-03T11:54:45.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.563+0000 7f0fb1925800 20 remove_watcher() i=4 2023-12-03T11:54:45.564 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.564+0000 7f0fb1925800 20 remove_watcher() i=5 2023-12-03T11:54:45.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.565+0000 7f0fb1925800 20 remove_watcher() i=6 2023-12-03T11:54:45.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.566+0000 7f0fb1925800 20 remove_watcher() i=7 2023-12-03T11:54:45.571 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-set-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.20 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:54:45.571 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:45.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.573 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.576 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.619 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.644 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with EIO injected at block_writes 2023-12-03T11:54:45.644 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:45.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.645 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.655 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.698 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.706 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.714 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.721 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.729 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.737 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.745 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.752 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.760 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.768 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.776 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.784 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.791 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.799 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.806 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.814 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.822 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.829 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.837 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:45.845 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:54:45.879 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:45.879 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:45.892 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.892+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.893+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 realm 2023-12-03T11:54:45.894 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.894+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.895+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.895+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.895+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:45.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.895+0000 7f3fca322800 20 searching for the correct realm 2023-12-03T11:54:45.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.904+0000 7f3fc5fa6640 0 -- 172.21.15.116:0/3706994552 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5567345dcf00 msgr2=0x5567345df2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.906+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 zone default found 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 4 Realm: () 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 zonegroup default 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.907+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.908+0000 7f3fca322800 20 started sync module instance, tier type = 2023-12-03T11:54:45.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.908+0000 7f3fca322800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:45.914 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.914+0000 7f3fc67a7640 0 -- 172.21.15.116:0/3706994552 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f3fb00093c0 msgr2=0x7f3fb009d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=0 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=1 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=2 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=3 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=4 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=5 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.916+0000 7f3fca322800 20 add_watcher() i=6 2023-12-03T11:54:45.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.917+0000 7f3fca322800 20 add_watcher() i=7 2023-12-03T11:54:45.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.917+0000 7f3fca322800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:45.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.918+0000 7f3f8b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.919+0000 7f3fca322800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:45.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.919+0000 7f3fca322800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:45.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.924+0000 7f3fca322800 5 note: GC not initialized 2023-12-03T11:54:45.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3efa7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:45.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3fca322800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:45.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3fca322800 20 Filter name: none 2023-12-03T11:54:45.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3eebfff640 20 reqs_thread_entry: start 2023-12-03T11:54:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.925+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.926+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.926+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:45.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.926+0000 7f3fca322800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:45.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.926+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.927+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:45.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.927+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.927+0000 7f3fca322800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:45.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.927+0000 7f3fca322800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:45.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.928+0000 7f3fca322800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.929+0000 7f3fca322800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.929+0000 7f3f8b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:45.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.931+0000 7f3fca322800 20 remove_watcher() i=0 2023-12-03T11:54:45.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.931+0000 7f3fca322800 2 removed watcher, disabling cache 2023-12-03T11:54:45.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.932+0000 7f3fca322800 20 remove_watcher() i=1 2023-12-03T11:54:45.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.933+0000 7f3fca322800 20 remove_watcher() i=2 2023-12-03T11:54:45.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.934+0000 7f3fca322800 20 remove_watcher() i=3 2023-12-03T11:54:45.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.935+0000 7f3fca322800 20 remove_watcher() i=4 2023-12-03T11:54:45.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.936+0000 7f3fca322800 20 remove_watcher() i=5 2023-12-03T11:54:45.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.937+0000 7f3fca322800 20 remove_watcher() i=6 2023-12-03T11:54:45.938 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.938+0000 7f3fca322800 20 remove_watcher() i=7 2023-12-03T11:54:45.944 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:45.944 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:45.944 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:45.944 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:45.945 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at block_writes 2023-12-03T11:54:45.977 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:45.977 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:45.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.990+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.991+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.991+0000 7f67fcf23800 20 realm 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.991+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.992+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.992+0000 7f67fcf23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.992+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.992+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:45.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.992+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.993+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:45.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.993+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:45.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.993+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:45.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:45.993+0000 7f67fcf23800 20 searching for the correct realm 2023-12-03T11:54:46.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.002+0000 7f67f8b3c640 0 -- 172.21.15.116:0/4017069969 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f67e00845d0 msgr2=0x7f67e0080a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.003+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.004 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.005 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 zone default found 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 4 Realm: () 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 zonegroup default 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.004+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.005+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.005+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.005+0000 7f67fcf23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.005+0000 7f67fcf23800 20 started sync module instance, tier type = 2023-12-03T11:54:46.006 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.005+0000 7f67fcf23800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.009+0000 7f67f8b3c640 0 -- 172.21.15.116:0/4017069969 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f67f4067030 msgr2=0x7f67f4071460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=0 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=1 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=2 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=3 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=4 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=5 2023-12-03T11:54:46.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.014+0000 7f67fcf23800 20 add_watcher() i=6 2023-12-03T11:54:46.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.015+0000 7f67fcf23800 20 add_watcher() i=7 2023-12-03T11:54:46.015 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.015+0000 7f67fcf23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:46.016 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.016+0000 7f67bf7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.017+0000 7f67fcf23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:46.017 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.017+0000 7f67fcf23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.022+0000 7f67fcf23800 5 note: GC not initialized 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.022+0000 7f672e7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.022+0000 7f67fcf23800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 20 Filter name: none 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f671ffff640 20 reqs_thread_entry: start 2023-12-03T11:54:46.023 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2023-12-03T11:54:46.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2023-12-03T11:54:46.024 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 : miss 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.023+0000 7f67fcf23800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 2023-12-03T11:54:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67fcf23800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67fcf23800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67fcf23800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67fcf23800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21, ofs0, ns] 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67eeb38640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660510 cookie 94634006365456 notifier 6323 bl.length()=877 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67eeb38640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67eeb38640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.029+0000 7f67eeb38640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_sources= new_sources= 2023-12-03T11:54:46.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential sources added=[] removed=[] 2023-12-03T11:54:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_dests= new_dests= 2023-12-03T11:54:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential dests added=[] removed=[] 2023-12-03T11:54:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 -1 Injecting error=-5 at location=block_writes 2023-12-03T11:54:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.030+0000 7f67fcf23800 0 ERROR: init_reshard failed to pause writes to the current index: (5) Input/output error 2023-12-03T11:54:46.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.032+0000 7f67fcf23800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:46.032 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.032+0000 7f67fcf23800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67fcf23800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67fcf23800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67fcf23800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67fcf23800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21, ofs0, ns] 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67eeb38640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660511 cookie 94634006365456 notifier 6323 bl.length()=845 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67eeb38640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67eeb38640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.033+0000 7f67eeb38640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.034+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_sources= new_sources= 2023-12-03T11:54:46.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.034+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential sources added=[] removed=[] 2023-12-03T11:54:46.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.034+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_dests= new_dests= 2023-12-03T11:54:46.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.034+0000 7f67fcf23800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential dests added=[] removed=[] 2023-12-03T11:54:46.036 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.035+0000 7f67bf7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.037+0000 7f67fcf23800 20 remove_watcher() i=0 2023-12-03T11:54:46.038 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.037+0000 7f67fcf23800 2 removed watcher, disabling cache 2023-12-03T11:54:46.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.039+0000 7f67fcf23800 20 remove_watcher() i=1 2023-12-03T11:54:46.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.040+0000 7f67fcf23800 20 remove_watcher() i=2 2023-12-03T11:54:46.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.041+0000 7f67fcf23800 20 remove_watcher() i=3 2023-12-03T11:54:46.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.042+0000 7f67fcf23800 20 remove_watcher() i=4 2023-12-03T11:54:46.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.043+0000 7f67fcf23800 20 remove_watcher() i=5 2023-12-03T11:54:46.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.044+0000 7f67fcf23800 20 remove_watcher() i=6 2023-12-03T11:54:46.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.044+0000 7f67fcf23800 20 remove_watcher() i=7 2023-12-03T11:54:46.051 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:54:46.078 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:46.078 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:46.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.091+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 realm 2023-12-03T11:54:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:46.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.093+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.094+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.094+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.094+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:46.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.094+0000 7f739d331800 20 searching for the correct realm 2023-12-03T11:54:46.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.103+0000 7f7398ffb640 0 -- 172.21.15.116:0/3492596536 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55ed90faa860 msgr2=0x55ed90facc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.104+0000 7f739d331800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.104+0000 7f739d331800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.104+0000 7f739d331800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.104+0000 7f739d331800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.104+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 zone default found 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 4 Realm: () 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 zonegroup default 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.105+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 started sync module instance, tier type = 2023-12-03T11:54:46.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.106+0000 7f739d331800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.113+0000 7f7393fff640 0 -- 172.21.15.116:0/3492596536 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55ed90faf870 msgr2=0x55ed90fb1c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:46.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=0 2023-12-03T11:54:46.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=1 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=2 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=3 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=4 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=5 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.115+0000 7f739d331800 20 add_watcher() i=6 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.116+0000 7f739d331800 20 add_watcher() i=7 2023-12-03T11:54:46.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.116+0000 7f739d331800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.118+0000 7f735e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.118+0000 7f739d331800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.118+0000 7f739d331800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f739d331800 5 note: GC not initialized 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f72cd7fa640 20 reqs_thread_entry: start 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f739d331800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f739d331800 20 Filter name: none 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f72c6ffd640 20 reqs_thread_entry: start 2023-12-03T11:54:46.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.124+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.125+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.125+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.125+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.125+0000 7f739d331800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.125+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.126+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.126+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.126+0000 7f739d331800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.126+0000 7f739d331800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.127+0000 7f739d331800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.127+0000 7f739d331800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.127+0000 7f735e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.129+0000 7f739d331800 20 remove_watcher() i=0 2023-12-03T11:54:46.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.129+0000 7f739d331800 2 removed watcher, disabling cache 2023-12-03T11:54:46.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.130+0000 7f739d331800 20 remove_watcher() i=1 2023-12-03T11:54:46.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.131+0000 7f739d331800 20 remove_watcher() i=2 2023-12-03T11:54:46.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.132+0000 7f739d331800 20 remove_watcher() i=3 2023-12-03T11:54:46.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.133+0000 7f739d331800 20 remove_watcher() i=4 2023-12-03T11:54:46.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.134+0000 7f739d331800 20 remove_watcher() i=5 2023-12-03T11:54:46.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.135+0000 7f739d331800 20 remove_watcher() i=6 2023-12-03T11:54:46.136 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.136+0000 7f739d331800 20 remove_watcher() i=7 2023-12-03T11:54:46.141 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.141 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:46.141 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:46.141 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.143 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.150 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.193 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:54:46.224 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:46.224 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:46.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.237+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.239 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 realm 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.239+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.240+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.240+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.241+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:46.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.241+0000 7f279e52e800 20 searching for the correct realm 2023-12-03T11:54:46.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.250+0000 7f279a2c6640 0 -- 172.21.15.116:0/1741304251 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55e26ac9be90 msgr2=0x55e26ac9e280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.251+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 zone default found 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 4 Realm: () 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 zonegroup default 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.252+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.253+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.253+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.253+0000 7f279e52e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.253+0000 7f279e52e800 20 started sync module instance, tier type = 2023-12-03T11:54:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.253+0000 7f279e52e800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.259+0000 7f2791ac5640 0 -- 172.21.15.116:0/1741304251 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55e26aca0e30 msgr2=0x55e26aca3220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=0 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=1 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=2 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=3 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=4 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=5 2023-12-03T11:54:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=6 2023-12-03T11:54:46.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 20 add_watcher() i=7 2023-12-03T11:54:46.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.262+0000 7f279e52e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.264+0000 7f275ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.265+0000 7f279e52e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.265+0000 7f279e52e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.270+0000 7f279e52e800 5 note: GC not initialized 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.270+0000 7f26ceffd640 20 reqs_thread_entry: start 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 20 Filter name: none 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f26ccff9640 20 reqs_thread_entry: start 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 : miss 2023-12-03T11:54:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.271+0000 7f279e52e800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.272+0000 7f279e52e800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 2023-12-03T11:54:46.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.276+0000 7f279e52e800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.276+0000 7f279e52e800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.276+0000 7f279e52e800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.276+0000 7f279e52e800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21, ofs0, ns] 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f277bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660512 cookie 94430942582864 notifier 6732 bl.length()=877 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f277bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f277bfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f277bfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_sources= new_sources= 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential sources added=[] removed=[] 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_dests= new_dests= 2023-12-03T11:54:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.277+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential dests added=[] removed=[] 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.0 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.0 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.1 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.1 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.2 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.2 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.3 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.3 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.4 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.4 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.5 2023-12-03T11:54:46.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.5 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.6 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.6 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.7 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.7 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.8 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.8 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.9 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.9 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.10 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.10 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.11 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1.11 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.279+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.0 2023-12-03T11:54:46.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.280+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.1 2023-12-03T11:54:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.280+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.2 2023-12-03T11:54:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.280+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.3 2023-12-03T11:54:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.281+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.4 2023-12-03T11:54:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.281+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.5 2023-12-03T11:54:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.282+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.6 2023-12-03T11:54:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.282+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.7 2023-12-03T11:54:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.283+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.8 2023-12-03T11:54:46.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.283+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.9 2023-12-03T11:54:46.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.283+0000 7f279e52e800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.21.10 2023-12-03T11:54:46.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.287+0000 7f279e52e800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 : hit (requested=0x17, cached=0x17) 2023-12-03T11:54:46.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.287+0000 7f279e52e800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 2023-12-03T11:54:46.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f279e52e800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f279e52e800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f279e52e800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f279e52e800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21, ofs0, ns] 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f2790ac3640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660513 cookie 94430942582864 notifier 6732 bl.length()=845 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f2790ac3640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 info.flags=0x17 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f2790ac3640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 to cache LRU end 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.288+0000 7f2790ac3640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.289+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_sources= new_sources= 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.289+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential sources added=[] removed=[] 2023-12-03T11:54:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.289+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): orig_dests= new_dests= 2023-12-03T11:54:46.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.289+0000 7f279e52e800 20 handle_bi_update(): bucket=:error-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.21]): potential dests added=[] removed=[] 2023-12-03T11:54:46.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.291+0000 7f279e52e800 1 execute INFO: reshard of bucket "error-at-block-writes" completed successfully 2023-12-03T11:54:46.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.292+0000 7f275ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.294+0000 7f279e52e800 20 remove_watcher() i=0 2023-12-03T11:54:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.294+0000 7f279e52e800 2 removed watcher, disabling cache 2023-12-03T11:54:46.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.295+0000 7f279e52e800 20 remove_watcher() i=1 2023-12-03T11:54:46.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.296+0000 7f279e52e800 20 remove_watcher() i=2 2023-12-03T11:54:46.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.297+0000 7f279e52e800 20 remove_watcher() i=3 2023-12-03T11:54:46.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.299+0000 7f279e52e800 20 remove_watcher() i=4 2023-12-03T11:54:46.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.300+0000 7f279e52e800 20 remove_watcher() i=5 2023-12-03T11:54:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.300+0000 7f279e52e800 20 remove_watcher() i=6 2023-12-03T11:54:46.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.301+0000 7f279e52e800 20 remove_watcher() i=7 2023-12-03T11:54:46.308 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:54:46.335 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:46.335 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:46.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.348+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 realm 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.350+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.351+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.351+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.351+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:46.352 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.351+0000 7f838a73a800 20 searching for the correct realm 2023-12-03T11:54:46.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.360+0000 7f8386406640 0 -- 172.21.15.116:0/2820537473 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f8380007c10 msgr2=0x7f838008ba00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.361+0000 7f838a73a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.361+0000 7f838a73a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.361+0000 7f838a73a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.361+0000 7f838a73a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.361+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 zone default found 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 4 Realm: () 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 zonegroup default 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.362+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.363+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.363+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.363+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.363+0000 7f838a73a800 20 started sync module instance, tier type = 2023-12-03T11:54:46.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.363+0000 7f838a73a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.368+0000 7f8386406640 0 -- 172.21.15.116:0/2820537473 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f8360079e40 msgr2=0x7f836007c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=0 2023-12-03T11:54:46.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=1 2023-12-03T11:54:46.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=2 2023-12-03T11:54:46.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=3 2023-12-03T11:54:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=4 2023-12-03T11:54:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=5 2023-12-03T11:54:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.372+0000 7f838a73a800 20 add_watcher() i=6 2023-12-03T11:54:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.373+0000 7f838a73a800 20 add_watcher() i=7 2023-12-03T11:54:46.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.373+0000 7f838a73a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:46.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.374+0000 7f834bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.375+0000 7f838a73a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:46.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.375+0000 7f838a73a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.380+0000 7f838a73a800 5 note: GC not initialized 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.380+0000 7f82baffd640 20 reqs_thread_entry: start 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.380+0000 7f838a73a800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.381+0000 7f838a73a800 20 Filter name: none 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.381+0000 7f82b8ff9640 20 reqs_thread_entry: start 2023-12-03T11:54:46.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.381+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.381+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.381+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.382+0000 7f838a73a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.383+0000 7f838a73a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.383+0000 7f838a73a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.384+0000 7f834bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.386+0000 7f838a73a800 20 remove_watcher() i=0 2023-12-03T11:54:46.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.386+0000 7f838a73a800 2 removed watcher, disabling cache 2023-12-03T11:54:46.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.387+0000 7f838a73a800 20 remove_watcher() i=1 2023-12-03T11:54:46.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.388+0000 7f838a73a800 20 remove_watcher() i=2 2023-12-03T11:54:46.389 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.389+0000 7f838a73a800 20 remove_watcher() i=3 2023-12-03T11:54:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.390+0000 7f838a73a800 20 remove_watcher() i=4 2023-12-03T11:54:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.391+0000 7f838a73a800 20 remove_watcher() i=5 2023-12-03T11:54:46.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.392+0000 7f838a73a800 20 remove_watcher() i=6 2023-12-03T11:54:46.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.393+0000 7f838a73a800 20 remove_watcher() i=7 2023-12-03T11:54:46.398 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.398 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:46.398 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.21 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:54:46.398 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.400 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.403 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.465 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.490 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with abort at block_writes 2023-12-03T11:54:46.490 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.491 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.502 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.545 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.552 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.560 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.569 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.577 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.585 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.592 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.600 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.607 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.615 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.623 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.630 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.638 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.645 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.653 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.661 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.668 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.675 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.684 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:46.692 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:54:46.726 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:46.726 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:46.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.738+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 20 realm 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.739+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.740+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.740+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.740+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.740+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.741+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:46.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.741+0000 7f6475b2f800 20 searching for the correct realm 2023-12-03T11:54:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.750+0000 7f64717d7640 0 -- 172.21.15.116:0/3204554684 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557540551da0 msgr2=0x557540554240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.751+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 zone default found 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 4 Realm: () 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 zonegroup default 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.752+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.753+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.753+0000 7f6475b2f800 20 started sync module instance, tier type = 2023-12-03T11:54:46.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.753+0000 7f6475b2f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.759+0000 7f6468fd6640 0 -- 172.21.15.116:0/3204554684 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x557540556e80 msgr2=0x557540559270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=0 2023-12-03T11:54:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=1 2023-12-03T11:54:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=2 2023-12-03T11:54:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=3 2023-12-03T11:54:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=4 2023-12-03T11:54:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=5 2023-12-03T11:54:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.762+0000 7f6475b2f800 20 add_watcher() i=6 2023-12-03T11:54:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.763+0000 7f6475b2f800 20 add_watcher() i=7 2023-12-03T11:54:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.763+0000 7f6475b2f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:46.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.764+0000 7f6436fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.764+0000 7f6475b2f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:46.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.764+0000 7f6475b2f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.770+0000 7f6475b2f800 5 note: GC not initialized 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.770+0000 7f63a5ffb640 20 reqs_thread_entry: start 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.770+0000 7f6475b2f800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.770+0000 7f6475b2f800 20 Filter name: none 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.770+0000 7f63977fe640 20 reqs_thread_entry: start 2023-12-03T11:54:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.771+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.771+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.771+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.772+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.772+0000 7f6475b2f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.772+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.772+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.772+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.773+0000 7f6475b2f800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.773+0000 7f6475b2f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:46.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.773+0000 7f6475b2f800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.774+0000 7f6475b2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.774+0000 7f6436fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:46.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.776+0000 7f6475b2f800 20 remove_watcher() i=0 2023-12-03T11:54:46.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.776+0000 7f6475b2f800 2 removed watcher, disabling cache 2023-12-03T11:54:46.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.777+0000 7f6475b2f800 20 remove_watcher() i=1 2023-12-03T11:54:46.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.778+0000 7f6475b2f800 20 remove_watcher() i=2 2023-12-03T11:54:46.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.779+0000 7f6475b2f800 20 remove_watcher() i=3 2023-12-03T11:54:46.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.780+0000 7f6475b2f800 20 remove_watcher() i=4 2023-12-03T11:54:46.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.781+0000 7f6475b2f800 20 remove_watcher() i=5 2023-12-03T11:54:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.782+0000 7f6475b2f800 20 remove_watcher() i=6 2023-12-03T11:54:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.783+0000 7f6475b2f800 20 remove_watcher() i=7 2023-12-03T11:54:46.789 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.789 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:46.789 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:46.789 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:46.791 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at block_writes 2023-12-03T11:54:46.822 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:46.823 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:46.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.835+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.836+0000 7f0e6613a800 20 realm 2023-12-03T11:54:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.837+0000 7f0e6613a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:46.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.838+0000 7f0e6613a800 20 searching for the correct realm 2023-12-03T11:54:46.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.847+0000 7f0e6160c640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55aa7fda5630 msgr2=0x55aa7fda7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:46.849 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:46.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zone default found 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Realm: () 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zonegroup default 2023-12-03T11:54:46.851 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started sync module instance, tier type = 2023-12-03T11:54:46.852 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:46.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.855+0000 7f0e6260e640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x55aa7fda5630 msgr2=0x7f0e3c07eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=0 2023-12-03T11:54:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=1 2023-12-03T11:54:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=2 2023-12-03T11:54:46.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=3 2023-12-03T11:54:47.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=4 2023-12-03T11:54:47.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=5 2023-12-03T11:54:47.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=6 2023-12-03T11:54:47.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=7 2023-12-03T11:54:47.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.057+0000 7f0e6613a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:47.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.059+0000 7f0e277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.065+0000 7f0e6613a800 5 note: GC not initialized 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.065+0000 7f0d967fc640 20 reqs_thread_entry: start 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.065+0000 7f0e6613a800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.065+0000 7f0e6613a800 20 Filter name: none 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0d87fff640 20 reqs_thread_entry: start 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:54:47.067 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:54:47.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:54:47.068 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : miss 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.066+0000 7f0e6613a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:54:47.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.072+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.072+0000 7f0e6613a800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.072+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.072+0000 7f0e6613a800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22, ofs0, ns] 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660559 cookie 94190777824944 notifier 6425 bl.length()=877 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_sources= new_sources= 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential sources added=[] removed=[] 2023-12-03T11:54:47.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_dests= new_dests= 2023-12-03T11:54:47.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential dests added=[] removed=[] 2023-12-03T11:54:47.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f0e6613a800 time 2023-12-03T11:54:47.075272+0000 2023-12-03T11:54:47.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f0e68d6eef8] 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.074+0000 7f0e6613a800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f0e6613a800 time 2023-12-03T11:54:47.075272+0000 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.092 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f0e68d6eef8] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f0e6613a800 thread_name:radosgw-admin 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0e66654db0] 2023-12-03T11:54:47.093 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f0e666a154c] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0e68d6ef52] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:54:47.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.075+0000 7f0e6613a800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f0e6613a800 thread_name:radosgw-admin 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0e66654db0] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f0e666a154c] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0e68d6ef52] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.095 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: dump: 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.096 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: dump: 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.097 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:54:46.829+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6419 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:54:46.829+0000 7f0e53fff640 4 set_mon_vals no callback set 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:54:46.829+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: get_monmap_and_config 2023-12-03T11:54:47.098 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:54:46.830+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.099 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:54:46.830+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.100 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.101 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:54:46.831+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:54:46.831+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:54:46.831+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x7ffda3c14458 EEXIST 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fd7e9c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fd8a260 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fd7eda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.137095s 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.102 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fd7e9c0 auth_method 0 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fd7eda0 auth_method 0 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.103 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6795 payload 274 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:54:46.831+0000 7f0e6260e640 1 monclient: found mon.noname-a 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e44069820 auth_method 0 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: dump: 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.104 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: dump: 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.105 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6795 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:54:46.832+0000 7f0e6613a800 4 set_mon_vals no callback set 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: shutdown 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.107 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:54:46.832+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 librados: starting msgr at 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 librados: starting objecter 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command objecter_requests cmddesc objecter_requests hook 0x55aa7fd372b0 EEXIST 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:54:46.833+0000 7f0e6613a800 1 librados: setting wanted keys 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:54:46.833+0000 7f0e6613a800 1 librados: calling monclient init 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.108 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:54:46.833+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:54:46.833+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.109 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x55aa7fd4e080 EEXIST 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fd7eda0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fd8a260 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fd8afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.139322s 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fd8afa0 auth_method 0 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.110 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fd8a260 auth_method 0 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fd7eda0 auth_method 0 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.111 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6425 payload 824 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:54:46.834+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:54:46.834+0000 7f0e6260e640 1 monclient: found mon.noname-c 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:54:46.834+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: dump: 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.112 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.113 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: dump: 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:54:46.834+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6425 2023-12-03T11:54:47.114 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:54:46.834+0000 7f0e43fff640 4 set_mon_vals no callback set 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:54:46.834+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:54:46.835+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:54:46.836+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e3c069850 auth_method 0 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:54:46.836+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fd9c8b0 auth_method 0 2023-12-03T11:54:47.115 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 realm 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:54:46.837+0000 7f0e6613a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:54:46.837+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x55aa7fda0630 auth_method 0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 searching for the correct realm 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:54:46.838+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.116 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:54:46.839+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e5801b3e0 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:54:46.839+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e5c08e020 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:54:46.840+0000 7f0e6160c640 10 monclient: get_auth_request con 0x7f0e54055900 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:54:46.842+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e5802d040 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:54:46.844+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e5803a950 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:54:46.847+0000 7f0e6160c640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55aa7fda5630 msgr2=0x55aa7fda7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:54:46.847+0000 7f0e427fc640 1 client.6425.objecter ms_handle_reset 0x55aa7fda5630 session 0x55aa7fb31060 osd.3 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:54:46.847+0000 7f0e427fc640 10 monclient: _renew_subs 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:54:46.847+0000 7f0e427fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:54:46.847+0000 7f0e6160c640 10 monclient: get_auth_request con 0x7f0e3c079e40 auth_method 0 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:47.117 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zone default found 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Realm: () 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:54:46.848+0000 7f0e6613a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zonegroup default 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.118 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started sync module instance, tier type = 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:54:46.855+0000 7f0e427fc640 1 client.6425.objecter ms_handle_reset 0x7f0e5802d040 session 0x7f0e5801bd40 osd.6 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:54:46.855+0000 7f0e6260e640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:54:46.855+0000 7f0e6260e640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x55aa7fda5630 msgr2=0x7f0e3c07eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=1 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=2 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=3 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:54:47.056+0000 7f0e6260e640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=4 2023-12-03T11:54:47.119 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=5 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=6 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=7 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:54:47.057+0000 7f0e6613a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.120 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:54:47.059+0000 7f0e277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:54:47.059+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache list hook 0x55aa7fd298b0 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache inspect hook 0x55aa7fd298b0 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache erase hook 0x55aa7fd298b0 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache zap hook 0x55aa7fd298b0 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: get_monmap_and_config 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.121 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:54:47.059+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.122 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.123 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.124 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x7ffda3c149d8 EEXIST 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fde29a0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fde1f80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fde3ef0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.125 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.366466s 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fde1f80 auth_method 0 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fde3ef0 auth_method 0 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.126 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fde29a0 auth_method 0 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_done global_id 6428 payload 274 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 1 monclient: found mon.noname-c 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: dump: 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: dump: 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.128 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:54:47.061+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6428 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:54:47.061+0000 7f0e6613a800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:54:47.061+0000 7f0e6613a800 4 set_mon_vals no callback set 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: shutdown 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.130 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:54:47.062+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: starting msgr at 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: starting objecter 2023-12-03T11:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command objecter_requests cmddesc objecter_requests hook 0x55aa7fdd83a0 EEXIST 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: setting wanted keys 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: calling monclient init 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.132 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:54:47.062+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:54:47.063+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x55aa7fde4ab0 EEXIST 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fde29a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fde3ef0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fdd5aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.133 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.368900s 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fde29a0 auth_method 0 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fdd5aa0 auth_method 0 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.134 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fde3ef0 auth_method 0 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6804 payload 824 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:54:47.063+0000 7f0e6260e640 1 monclient: found mon.noname-a 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: dump: 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.135 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: dump: 2023-12-03T11:54:47.136 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6804 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:54:47.063+0000 7f0e26fe5640 4 set_mon_vals no callback set 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.137 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:54:47.063+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cr dump hook 0x55aa7fdd6438 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 note: GC not initialized 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:54:47.065+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0d9c069850 auth_method 0 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:54:47.065+0000 7f0d967fc640 20 reqs_thread_entry: start 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace show hook 0x7f0e5c0047e0 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace history hook 0x7f0e5c0047e0 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace active hook 0x7f0e5c0047e0 2023-12-03T11:54:47.138 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace active_short hook 0x7f0e5c0047e0 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:54:47.065+0000 7f0e6613a800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:54:47.065+0000 7f0e6613a800 20 Filter name: none 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:54:47.066+0000 7f0d87fff640 20 reqs_thread_entry: start 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : miss 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.139 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:54:47.067+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fe31dc0 auth_method 0 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22, ofs0, ns] 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660559 cookie 94190777824944 notifier 6425 bl.length()=877 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_sources= new_sources= 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential sources added=[] removed=[] 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_dests= new_dests= 2023-12-03T11:54:47.140 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential dests added=[] removed=[] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:54:47.074+0000 7f0e6613a800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f0e6613a800 time 2023-12-03T11:54:47.075272+0000 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f0e68d6eef8] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:54:47.075+0000 7f0e6613a800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:47.141 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f0e6613a800 thread_name:radosgw-admin 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0e66654db0] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f0e666a154c] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0e68d6ef52] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.142 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:54:47.143 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:54:47.144 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:54:47.145 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:54:47.146 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:54:47.147 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:54:47.148 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0d87fff640 / http_manager 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0d967fc640 / http_manager 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e257e2640 / ms_dispatch 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e26fe5640 / io_context_pool 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e277e6640 / rgw_dt_lg_renew 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e427fc640 / ms_dispatch 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e43fff640 / io_context_pool 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e52ffd640 / ms_dispatch 2023-12-03T11:54:47.149 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e53fff640 / io_context_pool 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6160c640 / msgr-worker-2 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e61e0d640 / msgr-worker-1 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6260e640 / msgr-worker-0 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6613a800 / radosgw-admin 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/log/ceph/ceph-client.admin.102545.log 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: dump: 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.150 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:54:46.829+0000 7f0e52ffd640 10 monclient: dump: 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.151 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:54:46.829+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6419 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:54:46.829+0000 7f0e53fff640 4 set_mon_vals no callback set 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:54:46.829+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.152 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:54:46.829+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: get_monmap_and_config 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:54:46.830+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.153 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding auth protocol: cephx 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.154 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: crc 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fe90) adding con mode: secure 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:54:46.830+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:54:46.830+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding auth protocol: cephx 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.155 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: crc 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:54:46.830+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14310) adding con mode: secure 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:54:46.831+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:54:46.831+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:54:46.831+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x7ffda3c14458 EEXIST 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fd7e9c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fd8a260 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fd7eda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.156 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:54:46.831+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.137095s 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:54:46.831+0000 7f0e52ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fd7e9c0 auth_method 0 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fd7eda0 auth_method 0 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.157 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:54:46.831+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6795 payload 274 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:54:46.831+0000 7f0e6260e640 1 monclient: found mon.noname-a 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:54:46.831+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e44069820 auth_method 0 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: dump: 2023-12-03T11:54:47.158 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.159 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:54:46.832+0000 7f0e43fff640 10 monclient: dump: 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6795 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:47.160 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:54:46.832+0000 7f0e6613a800 4 set_mon_vals no callback set 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: shutdown 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:54:46.832+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.161 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding auth protocol: cephx 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: crc 2023-12-03T11:54:47.162 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:54:46.832+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4fd60) adding con mode: secure 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:54:46.832+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 librados: starting msgr at 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:54:46.832+0000 7f0e6613a800 1 librados: starting objecter 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command objecter_requests cmddesc objecter_requests hook 0x55aa7fd372b0 EEXIST 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:54:46.833+0000 7f0e6613a800 1 librados: setting wanted keys 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:54:46.833+0000 7f0e6613a800 1 librados: calling monclient init 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding auth protocol: cephx 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.163 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: crc 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fd4df38) adding con mode: secure 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:54:46.833+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:54:46.833+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:54:46.833+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x55aa7fd4e080 EEXIST 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:54:47.164 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fd7eda0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fd8a260 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fd8afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:54:46.833+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.139322s 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fd8afa0 auth_method 0 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fd8a260 auth_method 0 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.165 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fd7eda0 auth_method 0 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:54:46.833+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:54:46.833+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:54:46.833+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6425 payload 824 2023-12-03T11:54:47.166 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:54:46.834+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:54:46.834+0000 7f0e6260e640 1 monclient: found mon.noname-c 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:54:46.834+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: dump: 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.167 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:54:46.834+0000 7f0e427fc640 10 monclient: dump: 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.168 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:54:46.834+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6425 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:54:46.834+0000 7f0e43fff640 4 set_mon_vals no callback set 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:54:46.834+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:54:46.834+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:54:46.835+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:54:46.835+0000 7f0e427fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:54:46.836+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e3c069850 auth_method 0 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:54:46.836+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fd9c8b0 auth_method 0 2023-12-03T11:54:47.169 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 realm 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:54:46.836+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:54:46.837+0000 7f0e6613a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:54:46.837+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:54:46.837+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x55aa7fda0630 auth_method 0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:54:46.838+0000 7f0e6613a800 20 searching for the correct realm 2023-12-03T11:54:47.170 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:54:46.838+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:54:46.839+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e5801b3e0 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:54:46.839+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e5c08e020 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:54:46.840+0000 7f0e6160c640 10 monclient: get_auth_request con 0x7f0e54055900 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:54:46.842+0000 7f0e6260e640 10 monclient: get_auth_request con 0x7f0e5802d040 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:54:46.844+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0e5803a950 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:54:46.847+0000 7f0e6160c640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55aa7fda5630 msgr2=0x55aa7fda7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:54:46.847+0000 7f0e427fc640 1 client.6425.objecter ms_handle_reset 0x55aa7fda5630 session 0x55aa7fb31060 osd.3 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:54:46.847+0000 7f0e427fc640 10 monclient: _renew_subs 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:54:46.847+0000 7f0e427fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:54:46.847+0000 7f0e6160c640 10 monclient: get_auth_request con 0x7f0e3c079e40 auth_method 0 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:47.171 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zone default found 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Realm: () 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:54:46.848+0000 7f0e6613a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:54:46.848+0000 7f0e6613a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 zonegroup default 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:54:46.848+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.172 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started sync module instance, tier type = 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:54:46.849+0000 7f0e6613a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:54:46.855+0000 7f0e427fc640 1 client.6425.objecter ms_handle_reset 0x7f0e5802d040 session 0x7f0e5801bd40 osd.6 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:54:46.855+0000 7f0e6260e640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:54:46.855+0000 7f0e6260e640 0 -- 172.21.15.116:0/2373645500 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x55aa7fda5630 msgr2=0x7f0e3c07eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=1 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=2 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:54:46.858+0000 7f0e6613a800 20 add_watcher() i=3 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:54:47.056+0000 7f0e6260e640 10 monclient: get_auth_request con 0x55aa7fda5630 auth_method 0 2023-12-03T11:54:47.173 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=4 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=5 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=6 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:54:47.057+0000 7f0e6613a800 20 add_watcher() i=7 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:54:47.057+0000 7f0e6613a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding auth protocol: cephx 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.174 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:54:47.059+0000 7f0e277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: crc 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14d40) adding con mode: secure 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:54:47.059+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:54:47.059+0000 7f0e6613a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache list hook 0x55aa7fd298b0 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache inspect hook 0x55aa7fd298b0 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache erase hook 0x55aa7fd298b0 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cache zap hook 0x55aa7fd298b0 2023-12-03T11:54:47.175 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: get_monmap_and_config 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:54:47.059+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:54:47.059+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding auth protocol: cephx 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: crc 2023-12-03T11:54:47.177 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:54:47.059+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde1450) adding con mode: secure 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding auth protocol: cephx 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.178 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: crc 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 AuthRegistry(0x7ffda3c14890) adding con mode: secure 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:54:47.060+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:54:47.060+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x7ffda3c149d8 EEXIST 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fde29a0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fde1f80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fde3ef0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.179 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:54:47.060+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.366466s 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fde1f80 auth_method 0 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fde3ef0 auth_method 0 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:54:47.060+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:54:47.060+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.180 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:54:47.061+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fde29a0 auth_method 0 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:54:47.061+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient(hunting): handle_auth_done global_id 6428 payload 274 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 1 monclient: found mon.noname-c 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:54:47.061+0000 7f0e61e0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: dump: 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:54:47.182 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:54:47.061+0000 7f0e26fe5640 10 monclient: dump: 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:54:47.061+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6428 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:54:47.061+0000 7f0e6613a800 10 monclient: get_monmap_and_config success 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:54:47.061+0000 7f0e6613a800 4 set_mon_vals no callback set 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: shutdown 2023-12-03T11:54:47.183 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: build_initial_monmap 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 build_initial for_mkfs: 0 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: monmap: 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.184 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding auth protocol: cephx 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: crc 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fdd7c00) adding con mode: secure 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:54:47.062+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: starting msgr at 2023-12-03T11:54:47.185 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: starting objecter 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command objecter_requests cmddesc objecter_requests hook 0x55aa7fdd83a0 EEXIST 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: setting wanted keys 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:54:47.062+0000 7f0e6613a800 1 librados: calling monclient init 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:54:47.062+0000 7f0e6613a800 10 monclient: init 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding auth protocol: cephx 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.186 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: crc 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:54:47.062+0000 7f0e6613a800 5 AuthRegistry(0x55aa7fde4968) adding con mode: secure 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:54:47.062+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:54:47.063+0000 7f0e6613a800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command rotate-key cmddesc rotate-key hook 0x55aa7fde4ab0 EEXIST 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _reopen_session rank -1 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-a con 0x55aa7fde29a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-b con 0x55aa7fde3ef0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): picked mon.noname-c con 0x55aa7fdd5aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.187 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): start opening mon connection 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): _renew_subs 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient(hunting): authenticate will time out at 1543.368900s 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): get_auth_request con 0x55aa7fde29a0 auth_method 0 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): get_auth_request con 0x55aa7fdd5aa0 auth_method 0 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.188 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:54:47.063+0000 7f0e6160c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request con 0x55aa7fde3ef0 auth_method 0 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:54:47.063+0000 7f0e61e0d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient(hunting): handle_auth_done global_id 6804 payload 824 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient: _finish_hunting 0 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:54:47.063+0000 7f0e6260e640 1 monclient: found mon.noname-a 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:54:47.063+0000 7f0e6260e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:54:47.189 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: dump: 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.190 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: _finish_auth 0 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: _check_auth_tickets 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:54:47.063+0000 7f0e257e2640 10 monclient: dump: 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:54:47.191 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 monclient: authenticate success, global_id 6804 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:54:47.063+0000 7f0e26fe5640 4 set_mon_vals no callback set 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _renew_subs 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:54:47.063+0000 7f0e6613a800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:54:47.063+0000 7f0e6613a800 1 librados: init done 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:54:47.063+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command cr dump hook 0x55aa7fdd6438 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:54:47.065+0000 7f0e257e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 note: GC not initialized 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:54:47.065+0000 7f0e61e0d640 10 monclient: get_auth_request con 0x7f0d9c069850 auth_method 0 2023-12-03T11:54:47.192 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:54:47.065+0000 7f0d967fc640 20 reqs_thread_entry: start 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace show hook 0x7f0e5c0047e0 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace history hook 0x7f0e5c0047e0 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace active hook 0x7f0e5c0047e0 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:54:47.065+0000 7f0e6613a800 5 asok(0x55aa7fb44a10) register_command sync trace active_short hook 0x7f0e5c0047e0 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:54:47.065+0000 7f0e6613a800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:54:47.065+0000 7f0e6613a800 20 Filter name: none 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:54:47.066+0000 7f0d87fff640 20 reqs_thread_entry: start 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : miss 2023-12-03T11:54:47.193 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:54:47.066+0000 7f0e6613a800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:54:47.066+0000 7f0e6613a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:54:47.067+0000 7f0e6160c640 10 monclient: get_auth_request con 0x55aa7fe31dc0 auth_method 0 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:54:47.072+0000 7f0e6613a800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22, ofs0, ns] 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660559 cookie 94190777824944 notifier 6425 bl.length()=877 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.194 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:54:47.073+0000 7f0e43fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_sources= new_sources= 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential sources added=[] removed=[] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_dests= new_dests= 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:54:47.074+0000 7f0e6613a800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential dests added=[] removed=[] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:54:47.074+0000 7f0e6613a800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f0e6613a800 time 2023-12-03T11:54:47.075272+0000 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f0e68d6eef8] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:54:47.195 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:54:47.075+0000 7f0e6613a800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f0e6613a800 thread_name:radosgw-admin 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0e66654db0] 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f0e666a154c] 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f0e68d6ef52] 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f0e68d6f0b6] 2023-12-03T11:54:47.196 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55aa7eb94a71] 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x55aa7eba0ec4] 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f0e6663feb0] 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:54:47.197 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:54:47.198 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:54:47.199 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:54:47.200 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:54:47.201 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:54:47.202 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0d87fff640 / http_manager 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0d967fc640 / http_manager 2023-12-03T11:54:47.203 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e257e2640 / ms_dispatch 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e26fe5640 / io_context_pool 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e277e6640 / rgw_dt_lg_renew 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e427fc640 / ms_dispatch 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e43fff640 / io_context_pool 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e52ffd640 / ms_dispatch 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e53fff640 / io_context_pool 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6160c640 / msgr-worker-2 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e61e0d640 / msgr-worker-1 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6260e640 / msgr-worker-0 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: 7f0e6613a800 / radosgw-admin 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:54:47.076649Z_6b79b74c-93c7-4249-b483-e74dc9a6c038/log 2023-12-03T11:54:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:54:47.205 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.122+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 realm 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.124+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.125+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.125+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.125+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.125+0000 7f1129f2b800 20 searching for the correct realm 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.134+0000 7f1125c32640 0 -- 172.21.15.116:0/2086946681 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55e5f0fd4410 msgr2=0x55e5f0fd6800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.136+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 zone default found 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 4 Realm: () 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:47.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 zonegroup default 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.137+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 started sync module instance, tier type = 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.138+0000 7f1129f2b800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.143+0000 7f1126433640 0 -- 172.21.15.116:0/2086946681 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f111005edc0 msgr2=0x7f11100860c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=0 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=1 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=2 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=3 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=4 2023-12-03T11:54:47.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=5 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=6 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 20 add_watcher() i=7 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.147+0000 7f1129f2b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.149+0000 7f10eb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.149+0000 7f1129f2b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.149+0000 7f1129f2b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.155+0000 7f1129f2b800 5 note: GC not initialized 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.155+0000 7f105a7fc640 20 reqs_thread_entry: start 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.155+0000 7f1129f2b800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.155+0000 7f1129f2b800 20 Filter name: none 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.156+0000 7f104bfff640 20 reqs_thread_entry: start 2023-12-03T11:54:47.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.156+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.156+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.156+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.157+0000 7f1129f2b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.158+0000 7f1129f2b800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.158+0000 7f1129f2b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.159+0000 7f10eb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.161+0000 7f1129f2b800 20 remove_watcher() i=0 2023-12-03T11:54:47.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.161+0000 7f1129f2b800 2 removed watcher, disabling cache 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.162+0000 7f1129f2b800 20 remove_watcher() i=1 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.163+0000 7f1129f2b800 20 remove_watcher() i=2 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.164+0000 7f1129f2b800 20 remove_watcher() i=3 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.165+0000 7f1129f2b800 20 remove_watcher() i=4 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.166+0000 7f1129f2b800 20 remove_watcher() i=5 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.167+0000 7f1129f2b800 20 remove_watcher() i=6 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.168+0000 7f1129f2b800 20 remove_watcher() i=7 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:54:47.216 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:54:47.225 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:54:47.255 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:54:47.255 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:54:47.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.267+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 20 realm 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.268+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.269+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.269+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.270+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.270+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.270+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:54:47.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.270+0000 7f6aa3d35800 20 searching for the correct realm 2023-12-03T11:54:47.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.278+0000 7f6aa01df640 0 -- 172.21.15.116:0/1393056013 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x56412c4d9d50 msgr2=0x56412c4dc140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:54:47.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.280+0000 7f6aa3d35800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:54:47.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.280+0000 7f6aa3d35800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:54:47.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.280+0000 7f6aa3d35800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.280+0000 7f6aa3d35800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.280+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:54:47.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:54:47.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 zone default found 2023-12-03T11:54:47.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 4 Realm: () 2023-12-03T11:54:47.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:54:47.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 zonegroup default 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.281+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.282+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.282+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.282+0000 7f6aa3d35800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.282+0000 7f6aa3d35800 20 started sync module instance, tier type = 2023-12-03T11:54:47.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.282+0000 7f6aa3d35800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:54:47.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.286+0000 7f6aa01df640 0 -- 172.21.15.116:0/1393056013 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f6a88079040 msgr2=0x7f6a88078410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:54:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=0 2023-12-03T11:54:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=1 2023-12-03T11:54:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=2 2023-12-03T11:54:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=3 2023-12-03T11:54:47.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=4 2023-12-03T11:54:47.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=5 2023-12-03T11:54:47.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.291+0000 7f6aa3d35800 20 add_watcher() i=6 2023-12-03T11:54:47.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.292+0000 7f6aa3d35800 20 add_watcher() i=7 2023-12-03T11:54:47.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.292+0000 7f6aa3d35800 2 all 8 watchers are set, enabling cache 2023-12-03T11:54:47.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.293+0000 7f6a65fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.293+0000 7f6aa3d35800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:54:47.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.293+0000 7f6aa3d35800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:54:47.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.299+0000 7f6aa3d35800 5 note: GC not initialized 2023-12-03T11:54:47.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.299+0000 7f69dcff9640 20 reqs_thread_entry: start 2023-12-03T11:54:47.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.299+0000 7f6aa3d35800 20 init_complete bucket index max shards: 11 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.299+0000 7f6aa3d35800 20 Filter name: none 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f69d27fc640 20 reqs_thread_entry: start 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : miss 2023-12-03T11:54:47.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 20 rados->read ofs=0 len=0 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.300+0000 7f6aa3d35800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:54:47.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.302+0000 7f6aa3d35800 0 INFO: RGWReshardLock::lock found lock on abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to be held by another RGW process; skipping for now 2023-12-03T11:54:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.302+0000 7f6a65fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:54:47.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.304+0000 7f6aa3d35800 20 remove_watcher() i=0 2023-12-03T11:54:47.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.304+0000 7f6aa3d35800 2 removed watcher, disabling cache 2023-12-03T11:54:47.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.305+0000 7f6aa3d35800 20 remove_watcher() i=1 2023-12-03T11:54:47.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.307+0000 7f6aa3d35800 20 remove_watcher() i=2 2023-12-03T11:54:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.307+0000 7f6aa3d35800 20 remove_watcher() i=3 2023-12-03T11:54:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.308+0000 7f6aa3d35800 20 remove_watcher() i=4 2023-12-03T11:54:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.309+0000 7f6aa3d35800 20 remove_watcher() i=5 2023-12-03T11:54:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.310+0000 7f6aa3d35800 20 remove_watcher() i=6 2023-12-03T11:54:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:54:47.311+0000 7f6aa3d35800 20 remove_watcher() i=7 2023-12-03T11:54:47.317 INFO:tasks.workunit.client.0.smithi116.stderr:waiting 30 seconds for reshard lock to expire... 2023-12-03T11:55:17.333 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:55:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:17.382 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.380+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 20 realm 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.382+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.383+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.383+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.384+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.384+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.384+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:17.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.384+0000 7fa261613800 20 searching for the correct realm 2023-12-03T11:55:17.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.396+0000 7fa25f113640 0 -- 172.21.15.116:0/224803885 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55e9ae2f60a0 msgr2=0x55e9ae2f8490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.397+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 zone default found 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 4 Realm: () 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 zonegroup default 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.398+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.399+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.399+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.399+0000 7fa261613800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.399+0000 7fa261613800 20 started sync module instance, tier type = 2023-12-03T11:55:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.399+0000 7fa261613800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.407+0000 7fa25e912640 0 -- 172.21.15.116:0/224803885 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55e9ae2fb060 msgr2=0x55e9ae2fd450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=0 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=1 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=2 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=3 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=4 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=5 2023-12-03T11:55:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.409+0000 7fa261613800 20 add_watcher() i=6 2023-12-03T11:55:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.410+0000 7fa261613800 20 add_watcher() i=7 2023-12-03T11:55:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.410+0000 7fa261613800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.412+0000 7fa2257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:17.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.412+0000 7fa261613800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:17.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.412+0000 7fa261613800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:17.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa261613800 5 note: GC not initialized 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa193fff640 20 reqs_thread_entry: start 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa261613800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa261613800 20 Filter name: none 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa191ffb640 20 reqs_thread_entry: start 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa261613800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:55:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.419+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : miss 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:17.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.420+0000 7fa261613800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:55:17.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.422+0000 7fa261613800 10 init_target_layout removing existing target index objects from a previous reshard attempt 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.433+0000 7fa261613800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.433+0000 7fa261613800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.433+0000 7fa261613800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.433+0000 7fa261613800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22, ofs0, ns] 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa245ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660560 cookie 94462138114128 notifier 6861 bl.length()=877 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa245ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:55:17.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa245ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:55:17.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa245ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:17.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_sources= new_sources= 2023-12-03T11:55:17.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential sources added=[] removed=[] 2023-12-03T11:55:17.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_dests= new_dests= 2023-12-03T11:55:17.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.434+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential dests added=[] removed=[] 2023-12-03T11:55:17.438 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.0 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.0 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.1 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.1 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.2 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.2 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.3 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.3 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.4 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.4 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.5 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.5 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.6 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.6 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.7 2023-12-03T11:55:17.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.7 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.8 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.8 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.9 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.9 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.10 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.10 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.11 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2.11 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.0 2023-12-03T11:55:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.437+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.1 2023-12-03T11:55:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.438+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.2 2023-12-03T11:55:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.438+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.3 2023-12-03T11:55:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.439+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.4 2023-12-03T11:55:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.439+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.5 2023-12-03T11:55:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.440+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.6 2023-12-03T11:55:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.440+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.7 2023-12-03T11:55:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.441+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.8 2023-12-03T11:55:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.441+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.9 2023-12-03T11:55:17.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.442+0000 7fa261613800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.22.10 2023-12-03T11:55:17.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.445+0000 7fa261613800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:17.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.445+0000 7fa261613800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 2023-12-03T11:55:17.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa261613800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa261613800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa261613800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa261613800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22, ofs0, ns] 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa2457fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660561 cookie 94462138114128 notifier 6861 bl.length()=845 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa2457fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 info.flags=0x17 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa2457fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 to cache LRU end 2023-12-03T11:55:17.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.446+0000 7fa2457fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:17.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.447+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_sources= new_sources= 2023-12-03T11:55:17.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.447+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential sources added=[] removed=[] 2023-12-03T11:55:17.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.447+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): orig_dests= new_dests= 2023-12-03T11:55:17.449 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.447+0000 7fa261613800 20 handle_bi_update(): bucket=:abort-at-block-writes[21f2df23-11b5-4c91-821f-a8e55156195e.4668.22]): potential dests added=[] removed=[] 2023-12-03T11:55:17.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.449+0000 7fa261613800 1 execute INFO: reshard of bucket "abort-at-block-writes" completed successfully 2023-12-03T11:55:17.452 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.450+0000 7fa2257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.453+0000 7fa261613800 20 remove_watcher() i=0 2023-12-03T11:55:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.453+0000 7fa261613800 2 removed watcher, disabling cache 2023-12-03T11:55:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.454+0000 7fa261613800 20 remove_watcher() i=1 2023-12-03T11:55:17.456 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.455+0000 7fa261613800 20 remove_watcher() i=2 2023-12-03T11:55:17.457 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.456+0000 7fa261613800 20 remove_watcher() i=3 2023-12-03T11:55:17.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.457+0000 7fa261613800 20 remove_watcher() i=4 2023-12-03T11:55:17.459 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.458+0000 7fa261613800 20 remove_watcher() i=5 2023-12-03T11:55:17.460 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.459+0000 7fa261613800 20 remove_watcher() i=6 2023-12-03T11:55:17.461 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.460+0000 7fa261613800 20 remove_watcher() i=7 2023-12-03T11:55:17.467 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:55:17.495 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:17.495 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:17.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.509+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.510+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.510+0000 7f5af7537800 20 realm 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.510+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.511+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.511+0000 7f5af7537800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.511+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.511+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.511+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.512+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:17.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.512+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.512+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:17.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.512+0000 7f5af7537800 20 searching for the correct realm 2023-12-03T11:55:17.521 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.520+0000 7f5af3a3d640 0 -- 172.21.15.116:0/1151044669 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f5adc05bec0 msgr2=0x7f5adc05abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:17.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:17.524 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:17.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:17.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:17.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.525 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.523+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 zone default found 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 4 Realm: () 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 zonegroup default 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5aeaa3b640 0 -- 172.21.15.116:0/1151044669 >> [v2:172.21.15.152:6817/2654994651,v1:172.21.15.152:6819/2654994651] conn(0x7f5ae4028070 msgr2=0x7f5ae400f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.524+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.525+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.525+0000 7f5af7537800 20 started sync module instance, tier type = 2023-12-03T11:55:17.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.525+0000 7f5af7537800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=0 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=1 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=2 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=3 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=4 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=5 2023-12-03T11:55:17.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.533+0000 7f5af7537800 20 add_watcher() i=6 2023-12-03T11:55:17.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.534+0000 7f5af7537800 20 add_watcher() i=7 2023-12-03T11:55:17.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.534+0000 7f5af7537800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:17.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.536+0000 7f5ab97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:17.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.536+0000 7f5af7537800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:17.537 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.536+0000 7f5af7537800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5af7537800 5 note: GC not initialized 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5a27fff640 20 reqs_thread_entry: start 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5af7537800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5af7537800 20 Filter name: none 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5a25ffb640 20 reqs_thread_entry: start 2023-12-03T11:55:17.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.542+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.543+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:55:17.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.543+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:55:17.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:17.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:55:17.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:55:17.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.544+0000 7f5af7537800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:17.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.545+0000 7f5af7537800 20 rados->read ofs=0 len=0 2023-12-03T11:55:17.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.545+0000 7f5af7537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:17.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.546+0000 7f5ab97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:17.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.548+0000 7f5af7537800 20 remove_watcher() i=0 2023-12-03T11:55:17.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.548+0000 7f5af7537800 2 removed watcher, disabling cache 2023-12-03T11:55:17.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.549+0000 7f5af7537800 20 remove_watcher() i=1 2023-12-03T11:55:17.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.550+0000 7f5af7537800 20 remove_watcher() i=2 2023-12-03T11:55:17.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.551+0000 7f5af7537800 20 remove_watcher() i=3 2023-12-03T11:55:17.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.552+0000 7f5af7537800 20 remove_watcher() i=4 2023-12-03T11:55:17.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.553+0000 7f5af7537800 20 remove_watcher() i=5 2023-12-03T11:55:17.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.554+0000 7f5af7537800 20 remove_watcher() i=6 2023-12-03T11:55:17.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.555+0000 7f5af7537800 20 remove_watcher() i=7 2023-12-03T11:55:17.561 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:17.561 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:17.561 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-block-writes id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.22 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:55:17.561 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.563 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.566 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.609 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.635 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with EIO injected at commit_target_layout 2023-12-03T11:55:17.636 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:17.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.637 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.648 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.691 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.698 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.706 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.713 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.721 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.728 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.736 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.743 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.751 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.759 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.767 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.775 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.782 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.790 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.798 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.805 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.813 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.821 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.830 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:17.837 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:17.846 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:55:17.872 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:17.873 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:17.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:17.885+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 realm 2023-12-03T11:55:18.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:18.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.087+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.089+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.089+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.089+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:18.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.089+0000 7ff97a526800 20 searching for the correct realm 2023-12-03T11:55:18.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.097+0000 7ff976285640 0 -- 172.21.15.116:0/572387977 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x7ff950079e10 msgr2=0x7ff95007c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 zone default found 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 4 Realm: () 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 zonegroup default 2023-12-03T11:55:18.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.100+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 started sync module instance, tier type = 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.101+0000 7ff97a526800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:18.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.103+0000 7ff976a86640 0 -- 172.21.15.116:0/572387977 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7ff96c046040 msgr2=0x7ff96c019410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=0 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=1 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=2 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=3 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=4 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=5 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.110+0000 7ff97a526800 20 add_watcher() i=6 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.111+0000 7ff97a526800 20 add_watcher() i=7 2023-12-03T11:55:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.111+0000 7ff97a526800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:18.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.112+0000 7ff93bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.113+0000 7ff97a526800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:18.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.113+0000 7ff97a526800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.118+0000 7ff97a526800 5 note: GC not initialized 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.118+0000 7ff8aaffd640 20 reqs_thread_entry: start 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.118+0000 7ff97a526800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.118+0000 7ff97a526800 20 Filter name: none 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.119+0000 7ff8a8ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.119+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.119+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.119+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.120+0000 7ff97a526800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.121+0000 7ff97a526800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.121+0000 7ff97a526800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.122+0000 7ff93bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.125 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.124+0000 7ff97a526800 20 remove_watcher() i=0 2023-12-03T11:55:18.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.124+0000 7ff97a526800 2 removed watcher, disabling cache 2023-12-03T11:55:18.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.127+0000 7ff97a526800 20 remove_watcher() i=1 2023-12-03T11:55:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.128+0000 7ff97a526800 20 remove_watcher() i=2 2023-12-03T11:55:18.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.129+0000 7ff97a526800 20 remove_watcher() i=3 2023-12-03T11:55:18.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.129+0000 7ff97a526800 20 remove_watcher() i=4 2023-12-03T11:55:18.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.131+0000 7ff97a526800 20 remove_watcher() i=5 2023-12-03T11:55:18.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.131+0000 7ff97a526800 20 remove_watcher() i=6 2023-12-03T11:55:18.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.132+0000 7ff97a526800 20 remove_watcher() i=7 2023-12-03T11:55:18.138 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.138 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:18.138 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:18.139 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.140 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout 2023-12-03T11:55:18.172 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:18.173 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:18.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.185+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 realm 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.187+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.188+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.188+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.188+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:18.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.188+0000 7f75a4133800 20 searching for the correct realm 2023-12-03T11:55:18.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.197+0000 7f759fdfb640 0 -- 172.21.15.116:0/1703713867 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f7598024020 msgr2=0x7f7598031990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.199+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 zone default found 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 4 Realm: () 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 zonegroup default 2023-12-03T11:55:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 started sync module instance, tier type = 2023-12-03T11:55:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.200+0000 7f75a4133800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:18.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.203+0000 7f75a05fc640 0 -- 172.21.15.116:0/1703713867 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f75880030b0 msgr2=0x7f7594084400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:18.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=0 2023-12-03T11:55:18.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=1 2023-12-03T11:55:18.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=2 2023-12-03T11:55:18.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=3 2023-12-03T11:55:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=4 2023-12-03T11:55:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=5 2023-12-03T11:55:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.209+0000 7f75a4133800 20 add_watcher() i=6 2023-12-03T11:55:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.210+0000 7f75a4133800 20 add_watcher() i=7 2023-12-03T11:55:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.210+0000 7f75a4133800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:18.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.212+0000 7f7565fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.212+0000 7f75a4133800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:18.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.212+0000 7f75a4133800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.218+0000 7f75a4133800 5 note: GC not initialized 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.218+0000 7f74dcff9640 20 reqs_thread_entry: start 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.218+0000 7f75a4133800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.218+0000 7f75a4133800 20 Filter name: none 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.218+0000 7f74d27fc640 20 reqs_thread_entry: start 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:55:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 : miss 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.219+0000 7f75a4133800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 2023-12-03T11:55:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.224+0000 7f75a4133800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.224+0000 7f75a4133800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.224+0000 7f75a4133800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.224+0000 7f75a4133800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23, ofs0, ns] 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75867fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660528 cookie 94407414473840 notifier 6542 bl.length()=893 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75867fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75867fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75867fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75a4133800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_sources= new_sources= 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75a4133800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential sources added=[] removed=[] 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75a4133800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_dests= new_dests= 2023-12-03T11:55:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.225+0000 7f75a4133800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential dests added=[] removed=[] 2023-12-03T11:55:18.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.227+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.0 2023-12-03T11:55:18.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.0 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.1 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.1 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.2 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.2 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.3 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.3 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.4 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.4 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.5 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.5 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.6 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.6 2023-12-03T11:55:18.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.7 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.7 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.8 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.8 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.9 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.9 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.10 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.10 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.11 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1.11 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.0 2023-12-03T11:55:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.228+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.229+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.229+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.3 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.230+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.4 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.230+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.5 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.231+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.6 2023-12-03T11:55:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.231+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.7 2023-12-03T11:55:18.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.231+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.8 2023-12-03T11:55:18.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.232+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.9 2023-12-03T11:55:18.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.232+0000 7f75a4133800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.10 2023-12-03T11:55:18.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.235+0000 7f75a4133800 -1 Injecting error=-5 at location=commit_target_layout 2023-12-03T11:55:18.236 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.235+0000 7f75a4133800 0 ERROR: commit_reshard failed to commit target index layout: (5) Input/output error 2023-12-03T11:55:18.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.239+0000 7f7565fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.241+0000 7f75a4133800 20 remove_watcher() i=0 2023-12-03T11:55:18.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.241+0000 7f75a4133800 2 removed watcher, disabling cache 2023-12-03T11:55:18.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.242+0000 7f75a4133800 20 remove_watcher() i=1 2023-12-03T11:55:18.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.243+0000 7f75a4133800 20 remove_watcher() i=2 2023-12-03T11:55:18.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.244+0000 7f75a4133800 20 remove_watcher() i=3 2023-12-03T11:55:18.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.245+0000 7f75a4133800 20 remove_watcher() i=4 2023-12-03T11:55:18.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.246+0000 7f75a4133800 20 remove_watcher() i=5 2023-12-03T11:55:18.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.247+0000 7f75a4133800 20 remove_watcher() i=6 2023-12-03T11:55:18.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.248+0000 7f75a4133800 20 remove_watcher() i=7 2023-12-03T11:55:18.255 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:55:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:18.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.294+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 realm 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.296+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.297+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.297+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.297+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:18.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.297+0000 7f6226320800 20 searching for the correct realm 2023-12-03T11:55:18.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.307+0000 7f6221f1c640 0 -- 172.21.15.116:0/3796901367 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5604e777c610 msgr2=0x5604e777ea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.308+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 zone default found 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 4 Realm: () 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 zonegroup default 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.309+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.310+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.310+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.310+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.310+0000 7f6226320800 20 started sync module instance, tier type = 2023-12-03T11:55:18.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.310+0000 7f6226320800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:18.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.316+0000 7f622271d640 0 -- 172.21.15.116:0/3796901367 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f620c0a1b60 msgr2=0x7f620c0b3820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=0 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=1 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=2 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=3 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=4 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=5 2023-12-03T11:55:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.318+0000 7f6226320800 20 add_watcher() i=6 2023-12-03T11:55:18.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.319+0000 7f6226320800 20 add_watcher() i=7 2023-12-03T11:55:18.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.319+0000 7f6226320800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:18.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.321+0000 7f61e77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.321+0000 7f6226320800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:18.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.321+0000 7f6226320800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:18.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6226320800 5 note: GC not initialized 2023-12-03T11:55:18.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f61567fc640 20 reqs_thread_entry: start 2023-12-03T11:55:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6226320800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6226320800 20 Filter name: none 2023-12-03T11:55:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6147fff640 20 reqs_thread_entry: start 2023-12-03T11:55:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.327+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:55:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.328+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.329+0000 7f6226320800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.329+0000 7f6226320800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.330+0000 7f6226320800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.330+0000 7f6226320800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.330+0000 7f61e77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.333+0000 7f6226320800 20 remove_watcher() i=0 2023-12-03T11:55:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.333+0000 7f6226320800 2 removed watcher, disabling cache 2023-12-03T11:55:18.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.334+0000 7f6226320800 20 remove_watcher() i=1 2023-12-03T11:55:18.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.335+0000 7f6226320800 20 remove_watcher() i=2 2023-12-03T11:55:18.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.336+0000 7f6226320800 20 remove_watcher() i=3 2023-12-03T11:55:18.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.337+0000 7f6226320800 20 remove_watcher() i=4 2023-12-03T11:55:18.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.338+0000 7f6226320800 20 remove_watcher() i=5 2023-12-03T11:55:18.340 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.339+0000 7f6226320800 20 remove_watcher() i=6 2023-12-03T11:55:18.341 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.339+0000 7f6226320800 20 remove_watcher() i=7 2023-12-03T11:55:18.345 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.345 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:18.346 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:18.346 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.347 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.354 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.397 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:55:18.428 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:18.428 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:18.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.440+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 realm 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.442+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.443+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.443+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.443+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:18.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.443+0000 7f92c7b26800 20 searching for the correct realm 2023-12-03T11:55:18.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.653+0000 7f92c3861640 0 -- 172.21.15.116:0/25201579 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f92b80582e0 msgr2=0x7f92b8076210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.655+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 zone default found 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 4 Realm: () 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 zonegroup default 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.656+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.657+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.657+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.657+0000 7f92c7b26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.657+0000 7f92c7b26800 20 started sync module instance, tier type = 2023-12-03T11:55:18.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.657+0000 7f92c7b26800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:18.663 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.662+0000 7f92c4062640 0 -- 172.21.15.116:0/25201579 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f92b8052c20 msgr2=0x7f92b8056fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=0 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=1 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=2 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=3 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=4 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.666+0000 7f92c7b26800 20 add_watcher() i=5 2023-12-03T11:55:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.667+0000 7f92c7b26800 20 add_watcher() i=6 2023-12-03T11:55:18.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.667+0000 7f92c7b26800 20 add_watcher() i=7 2023-12-03T11:55:18.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.667+0000 7f92c7b26800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:18.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.668+0000 7f9289fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.669+0000 7f92c7b26800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:18.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.669+0000 7f92c7b26800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:18.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.674+0000 7f92c7b26800 5 note: GC not initialized 2023-12-03T11:55:18.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f9200ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:18.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:18.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 20 Filter name: none 2023-12-03T11:55:18.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f91f67fc640 20 reqs_thread_entry: start 2023-12-03T11:55:18.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:55:18.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 : miss 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.675+0000 7f92c7b26800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:18.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.676+0000 7f92c7b26800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 2023-12-03T11:55:18.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.677+0000 7f92c7b26800 10 init_target_layout removing existing target index objects from a previous reshard attempt 2023-12-03T11:55:18.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23, ofs0, ns] 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92aa7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660529 cookie 93845764095968 notifier 6975 bl.length()=893 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92aa7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92aa7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92aa7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_sources= new_sources= 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential sources added=[] removed=[] 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_dests= new_dests= 2023-12-03T11:55:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.683+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential dests added=[] removed=[] 2023-12-03T11:55:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.0 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.0 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.1 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.1 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.2 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.2 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.3 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.3 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.4 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.4 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.5 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.5 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.6 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.6 2023-12-03T11:55:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.7 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.7 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.8 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.8 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.9 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.9 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.10 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.10 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.11 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2.11 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.0 2023-12-03T11:55:18.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.686+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.1 2023-12-03T11:55:18.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.687+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.2 2023-12-03T11:55:18.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.687+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.3 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.688+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.4 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.688+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.5 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.688+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.6 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.689+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.7 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.689+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.8 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.690+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.9 2023-12-03T11:55:18.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.690+0000 7f92c7b26800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.23.10 2023-12-03T11:55:18.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.693+0000 7f92c7b26800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:18.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.693+0000 7f92c7b26800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 2023-12-03T11:55:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92c7b26800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92c7b26800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92c7b26800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92c7b26800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23, ofs0, ns] 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92a9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660530 cookie 93845764095968 notifier 6975 bl.length()=861 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92a9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 info.flags=0x17 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92a9ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 to cache LRU end 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.694+0000 7f92a9ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.695+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_sources= new_sources= 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.695+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential sources added=[] removed=[] 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.695+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): orig_dests= new_dests= 2023-12-03T11:55:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.695+0000 7f92c7b26800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.23]): potential dests added=[] removed=[] 2023-12-03T11:55:18.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.697+0000 7f92c7b26800 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2023-12-03T11:55:18.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.699+0000 7f9289fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.701+0000 7f92c7b26800 20 remove_watcher() i=0 2023-12-03T11:55:18.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.701+0000 7f92c7b26800 2 removed watcher, disabling cache 2023-12-03T11:55:18.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.702+0000 7f92c7b26800 20 remove_watcher() i=1 2023-12-03T11:55:18.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.703+0000 7f92c7b26800 20 remove_watcher() i=2 2023-12-03T11:55:18.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.704+0000 7f92c7b26800 20 remove_watcher() i=3 2023-12-03T11:55:18.706 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.705+0000 7f92c7b26800 20 remove_watcher() i=4 2023-12-03T11:55:18.707 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.706+0000 7f92c7b26800 20 remove_watcher() i=5 2023-12-03T11:55:18.708 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.707+0000 7f92c7b26800 20 remove_watcher() i=6 2023-12-03T11:55:18.709 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.708+0000 7f92c7b26800 20 remove_watcher() i=7 2023-12-03T11:55:18.715 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:55:18.742 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:18.742 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:18.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.755+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 realm 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.756+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.757+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.757+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.758+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:18.759 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.758+0000 7fdde240f800 20 searching for the correct realm 2023-12-03T11:55:18.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.766+0000 7fdddda8c640 0 -- 172.21.15.116:0/1487917958 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fddd802b070 msgr2=0x7fddd8057cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:18.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 20 zone default found 2023-12-03T11:55:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.768+0000 7fdde240f800 4 Realm: () 2023-12-03T11:55:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 zonegroup default 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 started sync module instance, tier type = 2023-12-03T11:55:18.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.769+0000 7fdde240f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:18.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.774+0000 7fdddda8c640 0 -- 172.21.15.116:0/1487917958 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fddb8079d90 msgr2=0x7fddb807c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=0 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=1 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=2 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=3 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=4 2023-12-03T11:55:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=5 2023-12-03T11:55:18.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=6 2023-12-03T11:55:18.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 20 add_watcher() i=7 2023-12-03T11:55:18.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.778+0000 7fdde240f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:18.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.780+0000 7fdda37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.780+0000 7fdde240f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:18.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.780+0000 7fdde240f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.786+0000 7fdde240f800 5 note: GC not initialized 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.786+0000 7fdd127fc640 20 reqs_thread_entry: start 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.786+0000 7fdde240f800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.787+0000 7fdde240f800 20 Filter name: none 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.787+0000 7fdd03fff640 20 reqs_thread_entry: start 2023-12-03T11:55:18.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.787+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.787+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.787+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:18.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:18.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:18.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.788+0000 7fdde240f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:18.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.789+0000 7fdde240f800 20 rados->read ofs=0 len=0 2023-12-03T11:55:18.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.790+0000 7fdde240f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:18.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.790+0000 7fdda37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:18.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.792+0000 7fdde240f800 20 remove_watcher() i=0 2023-12-03T11:55:18.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.792+0000 7fdde240f800 2 removed watcher, disabling cache 2023-12-03T11:55:18.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.794+0000 7fdde240f800 20 remove_watcher() i=1 2023-12-03T11:55:18.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.795+0000 7fdde240f800 20 remove_watcher() i=2 2023-12-03T11:55:18.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.796+0000 7fdde240f800 20 remove_watcher() i=3 2023-12-03T11:55:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.797+0000 7fdde240f800 20 remove_watcher() i=4 2023-12-03T11:55:18.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.798+0000 7fdde240f800 20 remove_watcher() i=5 2023-12-03T11:55:18.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.799+0000 7fdde240f800 20 remove_watcher() i=6 2023-12-03T11:55:18.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:18.800+0000 7fdde240f800 20 remove_watcher() i=7 2023-12-03T11:55:18.806 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.806 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:18.806 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.23 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:55:18.806 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.808 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.811 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.874 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.901 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with ECANCELED injected at commit_target_layout 2023-12-03T11:55:18.902 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:18.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.902 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.914 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.956 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.964 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.970 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.978 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.985 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:18.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:18.993 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.001 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.008 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.015 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.022 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.029 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.036 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.043 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.050 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.057 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.065 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.072 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.080 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.087 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.094 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.103 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:55:19.130 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:19.130 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:19.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.142+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 realm 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.144+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.145+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.145+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.145+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.145+0000 7f5ca7d31800 20 searching for the correct realm 2023-12-03T11:55:19.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.154+0000 7f5ca39fb640 0 -- 172.21.15.116:0/1675923620 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x56148395b5c0 msgr2=0x56148395d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:19.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.156+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 zone default found 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 4 Realm: () 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 zonegroup default 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.157+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.158+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.158+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.158+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.158+0000 7f5ca7d31800 20 started sync module instance, tier type = 2023-12-03T11:55:19.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.158+0000 7f5ca7d31800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.164 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.163+0000 7f5ca41fc640 0 -- 172.21.15.116:0/1675923620 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f5c8c058780 msgr2=0x7f5c8c00da90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=0 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=1 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=2 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=3 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=4 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=5 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.166+0000 7f5ca7d31800 20 add_watcher() i=6 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.167+0000 7f5ca7d31800 20 add_watcher() i=7 2023-12-03T11:55:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.167+0000 7f5ca7d31800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.169+0000 7f5c69fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.169+0000 7f5ca7d31800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.169+0000 7f5ca7d31800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.175+0000 7f5ca7d31800 5 note: GC not initialized 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.175+0000 7f5be0ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.175+0000 7f5ca7d31800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.175+0000 7f5ca7d31800 20 Filter name: none 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.175+0000 7f5bd67fc640 20 reqs_thread_entry: start 2023-12-03T11:55:19.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.176+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.178 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.176+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.176+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.177+0000 7f5ca7d31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.178+0000 7f5ca7d31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.178+0000 7f5ca7d31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.179+0000 7f5c69fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.181+0000 7f5ca7d31800 20 remove_watcher() i=0 2023-12-03T11:55:19.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.181+0000 7f5ca7d31800 2 removed watcher, disabling cache 2023-12-03T11:55:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.182+0000 7f5ca7d31800 20 remove_watcher() i=1 2023-12-03T11:55:19.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.183+0000 7f5ca7d31800 20 remove_watcher() i=2 2023-12-03T11:55:19.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.184+0000 7f5ca7d31800 20 remove_watcher() i=3 2023-12-03T11:55:19.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.185+0000 7f5ca7d31800 20 remove_watcher() i=4 2023-12-03T11:55:19.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.186+0000 7f5ca7d31800 20 remove_watcher() i=5 2023-12-03T11:55:19.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.187+0000 7f5ca7d31800 20 remove_watcher() i=6 2023-12-03T11:55:19.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.188+0000 7f5ca7d31800 20 remove_watcher() i=7 2023-12-03T11:55:19.194 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.194 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:19.194 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:19.194 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.195 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout --inject-error-code 125 2023-12-03T11:55:19.229 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:19.229 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:19.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.241+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 realm 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.243+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.244+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.244+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.244+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.244+0000 7f0863f46800 20 searching for the correct realm 2023-12-03T11:55:19.255 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.254+0000 7f085fd0a640 0 -- 172.21.15.116:0/3472889646 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55a93b80c450 msgr2=0x55a93b80e860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.255+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 zone default found 2023-12-03T11:55:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 4 Realm: () 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 zonegroup default 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.256+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.257+0000 7f0863f46800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.257+0000 7f0863f46800 20 started sync module instance, tier type = 2023-12-03T11:55:19.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.257+0000 7f0863f46800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.262+0000 7f086050b640 0 -- 172.21.15.116:0/3472889646 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f084803f040 msgr2=0x7f084803f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=0 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=1 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=2 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=3 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=4 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=5 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.265+0000 7f0863f46800 20 add_watcher() i=6 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.266+0000 7f0863f46800 20 add_watcher() i=7 2023-12-03T11:55:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.266+0000 7f0863f46800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.267+0000 7f0825fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.268+0000 7f0863f46800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.268+0000 7f0863f46800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.273+0000 7f0863f46800 5 note: GC not initialized 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.273+0000 7f079cff9640 20 reqs_thread_entry: start 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.273+0000 7f0863f46800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.273+0000 7f0863f46800 20 Filter name: none 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f07927fc640 20 reqs_thread_entry: start 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:55:19.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:19.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 : miss 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 info.flags=0x17 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 to cache LRU end 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.274+0000 7f0863f46800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 2023-12-03T11:55:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0863f46800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 info.flags=0x17 2023-12-03T11:55:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0863f46800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 to cache LRU end 2023-12-03T11:55:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0863f46800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0863f46800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24, ofs0, ns] 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0855d06640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660533 cookie 94185336173376 notifier 7041 bl.length()=893 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0855d06640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 info.flags=0x17 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0855d06640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 to cache LRU end 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.280+0000 7f0855d06640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.281+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): orig_sources= new_sources= 2023-12-03T11:55:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.281+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): potential sources added=[] removed=[] 2023-12-03T11:55:19.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.281+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): orig_dests= new_dests= 2023-12-03T11:55:19.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.281+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): potential dests added=[] removed=[] 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.0 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.0 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.1 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.1 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.2 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.2 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.3 2023-12-03T11:55:19.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.3 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.4 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.4 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.5 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.5 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.6 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.6 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.7 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.7 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.8 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.8 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.9 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.9 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.10 2023-12-03T11:55:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.10 2023-12-03T11:55:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.11 2023-12-03T11:55:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1.11 2023-12-03T11:55:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.285+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.0 2023-12-03T11:55:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.286+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.1 2023-12-03T11:55:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.286+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.2 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.287+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.3 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.287+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.4 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.288+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.5 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.288+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.6 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.289+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.7 2023-12-03T11:55:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.289+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.8 2023-12-03T11:55:19.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.290+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.9 2023-12-03T11:55:19.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.290+0000 7f0863f46800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.24.10 2023-12-03T11:55:19.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.295+0000 7f0863f46800 -1 Injecting error=-125 at location=commit_target_layout 2023-12-03T11:55:19.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.295+0000 7f0863f46800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:19.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.295+0000 7f0863f46800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 2023-12-03T11:55:19.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 info.flags=0x17 2023-12-03T11:55:19.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 to cache LRU end 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24, ofs0, ns] 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0856507640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660534 cookie 94185336173376 notifier 7041 bl.length()=861 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0856507640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 info.flags=0x17 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0856507640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 to cache LRU end 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0856507640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): orig_sources= new_sources= 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): potential sources added=[] removed=[] 2023-12-03T11:55:19.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): orig_dests= new_dests= 2023-12-03T11:55:19.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.296+0000 7f0863f46800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.24]): potential dests added=[] removed=[] 2023-12-03T11:55:19.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.301+0000 7f0863f46800 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2023-12-03T11:55:19.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.303+0000 7f0825fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.305+0000 7f0863f46800 20 remove_watcher() i=0 2023-12-03T11:55:19.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.305+0000 7f0863f46800 2 removed watcher, disabling cache 2023-12-03T11:55:19.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.306+0000 7f0863f46800 20 remove_watcher() i=1 2023-12-03T11:55:19.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.307+0000 7f0863f46800 20 remove_watcher() i=2 2023-12-03T11:55:19.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.308+0000 7f0863f46800 20 remove_watcher() i=3 2023-12-03T11:55:19.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.309+0000 7f0863f46800 20 remove_watcher() i=4 2023-12-03T11:55:19.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.310+0000 7f0863f46800 20 remove_watcher() i=5 2023-12-03T11:55:19.312 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.311+0000 7f0863f46800 20 remove_watcher() i=6 2023-12-03T11:55:19.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.312+0000 7f0863f46800 20 remove_watcher() i=7 2023-12-03T11:55:19.319 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:55:19.346 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:19.346 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:19.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.358+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 realm 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.360+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.361+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.361+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.361+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.361+0000 7f1d72337800 20 searching for the correct realm 2023-12-03T11:55:19.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.369+0000 7f1d6e03c640 0 -- 172.21.15.116:0/60097726 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1d6801a400 msgr2=0x7f1d6801a820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.371+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 zone default found 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 4 Realm: () 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 zonegroup default 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.372+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.373+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.373+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.373+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.373+0000 7f1d72337800 20 started sync module instance, tier type = 2023-12-03T11:55:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.373+0000 7f1d72337800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.377 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.375+0000 7f1d6e03c640 0 -- 172.21.15.116:0/60097726 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55c357d00670 msgr2=0x55c357d02a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=0 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=1 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=2 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=3 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=4 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=5 2023-12-03T11:55:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.381+0000 7f1d72337800 20 add_watcher() i=6 2023-12-03T11:55:19.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.382+0000 7f1d72337800 20 add_watcher() i=7 2023-12-03T11:55:19.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.382+0000 7f1d72337800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.384+0000 7f1d33fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.384+0000 7f1d72337800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.384+0000 7f1d72337800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1d72337800 5 note: GC not initialized 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1ca2ffd640 20 reqs_thread_entry: start 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1d72337800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1d72337800 20 Filter name: none 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1ca0ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.390+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.391+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.391+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.391+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.391+0000 7f1d72337800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.391+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.392+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.392+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.392+0000 7f1d72337800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.392+0000 7f1d72337800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.393+0000 7f1d72337800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.393+0000 7f1d72337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.394+0000 7f1d33fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.396+0000 7f1d72337800 20 remove_watcher() i=0 2023-12-03T11:55:19.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.396+0000 7f1d72337800 2 removed watcher, disabling cache 2023-12-03T11:55:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.397+0000 7f1d72337800 20 remove_watcher() i=1 2023-12-03T11:55:19.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.397+0000 7f1d72337800 20 remove_watcher() i=2 2023-12-03T11:55:19.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.398+0000 7f1d72337800 20 remove_watcher() i=3 2023-12-03T11:55:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.399+0000 7f1d72337800 20 remove_watcher() i=4 2023-12-03T11:55:19.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.401+0000 7f1d72337800 20 remove_watcher() i=5 2023-12-03T11:55:19.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.401+0000 7f1d72337800 20 remove_watcher() i=6 2023-12-03T11:55:19.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.402+0000 7f1d72337800 20 remove_watcher() i=7 2023-12-03T11:55:19.409 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.409 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:19.409 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.24 num_objects: 20 size_kb: 1 num_shards: 12 2023-12-03T11:55:19.410 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.410 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.413 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.456 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.482 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with abort at commit_target_layout 2023-12-03T11:55:19.482 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.483 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.527 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.570 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.578 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.586 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.594 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.602 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.609 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.617 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.624 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.632 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.640 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.647 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.655 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.662 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.670 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.678 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.686 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.694 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.702 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.710 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:19.717 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:19.725 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:55:19.752 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:19.752 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:19.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.764+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 realm 2023-12-03T11:55:19.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.766+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.767+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.767+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.767+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.767+0000 7ff78128c800 20 searching for the correct realm 2023-12-03T11:55:19.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.777+0000 7ff77ed8c640 0 -- 172.21.15.116:0/3623043125 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55d527ba90e0 msgr2=0x55d527bab4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 zone default found 2023-12-03T11:55:19.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 4 Realm: () 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 zonegroup default 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.778+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 started sync module instance, tier type = 2023-12-03T11:55:19.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.779+0000 7ff78128c800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.786+0000 7ff77f58d640 0 -- 172.21.15.116:0/3623043125 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7ff768035040 msgr2=0x7ff768036940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=0 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=1 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=2 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=3 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=4 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=5 2023-12-03T11:55:19.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.788+0000 7ff78128c800 20 add_watcher() i=6 2023-12-03T11:55:19.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.789+0000 7ff78128c800 20 add_watcher() i=7 2023-12-03T11:55:19.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.789+0000 7ff78128c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.790+0000 7ff6bffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.790+0000 7ff78128c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.790+0000 7ff78128c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff78128c800 5 note: GC not initialized 2023-12-03T11:55:19.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff6aeffd640 20 reqs_thread_entry: start 2023-12-03T11:55:19.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff78128c800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff78128c800 20 Filter name: none 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff6acff9640 20 reqs_thread_entry: start 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.796+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.797+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.797+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.797+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.797+0000 7ff78128c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.797+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.798+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.798+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.798+0000 7ff78128c800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.798+0000 7ff78128c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:19.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.799+0000 7ff78128c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.799+0000 7ff78128c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.800+0000 7ff6bffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.803+0000 7ff78128c800 20 remove_watcher() i=0 2023-12-03T11:55:19.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.803+0000 7ff78128c800 2 removed watcher, disabling cache 2023-12-03T11:55:19.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.804+0000 7ff78128c800 20 remove_watcher() i=1 2023-12-03T11:55:19.806 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.805+0000 7ff78128c800 20 remove_watcher() i=2 2023-12-03T11:55:19.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.806+0000 7ff78128c800 20 remove_watcher() i=3 2023-12-03T11:55:19.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.807+0000 7ff78128c800 20 remove_watcher() i=4 2023-12-03T11:55:19.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.808+0000 7ff78128c800 20 remove_watcher() i=5 2023-12-03T11:55:19.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.808+0000 7ff78128c800 20 remove_watcher() i=6 2023-12-03T11:55:19.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.809+0000 7ff78128c800 20 remove_watcher() i=7 2023-12-03T11:55:19.817 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.818 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:19.818 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:19.818 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.818 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at commit_target_layout 2023-12-03T11:55:19.849 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:19.849 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:19.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.861+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 realm 2023-12-03T11:55:19.864 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.865 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.866 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.864+0000 7f190fdaf800 20 searching for the correct realm 2023-12-03T11:55:19.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.873+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557795e67bf0 msgr2=0x557795e69fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zone default found 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Realm: () 2023-12-03T11:55:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zonegroup default 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started sync module instance, tier type = 2023-12-03T11:55:19.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.882+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1908027900 msgr2=0x7f1908028410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=0 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=1 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=2 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=3 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=4 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=5 2023-12-03T11:55:19.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=6 2023-12-03T11:55:19.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=7 2023-12-03T11:55:19.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.885+0000 7f190fdaf800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.887+0000 7f18d37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.889 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.893+0000 7f190fdaf800 5 note: GC not initialized 2023-12-03T11:55:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.893+0000 7f18427fc640 20 reqs_thread_entry: start 2023-12-03T11:55:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 Filter name: none 2023-12-03T11:55:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f1833fff640 20 reqs_thread_entry: start 2023-12-03T11:55:19.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:55:19.896 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 : miss 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.895+0000 7f190fdaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 2023-12-03T11:55:19.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.900+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.900+0000 7f190fdaf800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.900+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.900+0000 7f190fdaf800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25, ofs0, ns] 2023-12-03T11:55:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660538 cookie 93972104423024 notifier 6701 bl.length()=893 2023-12-03T11:55:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_sources= new_sources= 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential sources added=[] removed=[] 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_dests= new_dests= 2023-12-03T11:55:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential dests added=[] removed=[] 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.0 2023-12-03T11:55:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.2 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.3 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.4 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.5 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.6 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.7 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.8 2023-12-03T11:55:19.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.9 2023-12-03T11:55:19.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.10 2023-12-03T11:55:19.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f190fdaf800 time 2023-12-03T11:55:19.911578+0000 2023-12-03T11:55:19.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f191476eef8] 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.912 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f190fdaf800 time 2023-12-03T11:55:19.911578+0000 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f191476eef8] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:19.913 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.914 INFO:tasks.workunit.client.0.smithi116.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:55:19.914 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f190fdaf800 thread_name:radosgw-admin 2023-12-03T11:55:19.933 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.933 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1912054db0] 2023-12-03T11:55:19.933 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f19120a154c] 2023-12-03T11:55:19.933 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:19.933 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f191476ef52] 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f190fdaf800 thread_name:radosgw-admin 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.934 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1912054db0] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f19120a154c] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f191476ef52] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:19.935 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.936 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:55:19.857+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:55:19.857+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.937 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0af60918 EEXIST 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e4c060 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795e47ff0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e4c440 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.163663s 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e47ff0 auth_method 0 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.938 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e4c060 auth_method 0 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e4c440 auth_method 0 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.939 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_done global_id 7101 payload 274 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient: _finish_hunting 0 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:55:19.858+0000 7f190d8af640 1 monclient: found mon.noname-a 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f0069820 auth_method 0 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: dump: 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.940 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: _finish_auth 0 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: _check_auth_tickets 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: dump: 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:55:19.858+0000 7f190fdaf800 5 monclient: authenticate success, global_id 7101 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:55:19.858+0000 7f190fdaf800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:55:19.858+0000 7f190fdaf800 4 set_mon_vals no callback set 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: shutdown 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:19.942 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:19.943 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:55:19.859+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: starting msgr at 2023-12-03T11:55:19.944 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: starting objecter 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command objecter_requests cmddesc objecter_requests hook 0x557795e1bb80 EEXIST 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: setting wanted keys 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: calling monclient init 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:19.945 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:19.946 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:55:19.859+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:55:19.860+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:55:19.860+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x557795d52220 EEXIST 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e47ff0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795e4c060 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e55240 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.947 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.165801s 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e47ff0 auth_method 0 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e4c060 auth_method 0 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.948 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e55240 auth_method 0 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.949 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_done global_id 6701 payload 824 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient: _finish_hunting 0 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:55:19.860+0000 7f190d8af640 1 monclient: found mon.noname-c 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: dump: 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.950 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: _finish_auth 0 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: _check_auth_tickets 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: dump: 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.951 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:55:19.860+0000 7f190fdaf800 5 monclient: authenticate success, global_id 6701 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:55:19.860+0000 7f18effff640 4 set_mon_vals no callback set 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:55:19.860+0000 7f190fdaf800 1 librados: init done 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:55:19.861+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.952 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:55:19.862+0000 7f19050ae640 10 monclient: get_auth_request con 0x7f18e8069850 auth_method 0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:55:19.862+0000 7f190e0b0640 10 monclient: get_auth_request con 0x557795e63970 auth_method 0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 realm 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:55:19.863+0000 7f190fdaf800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:55:19.863+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e67bf0 auth_method 0 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.953 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 searching for the correct realm 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:55:19.864+0000 7f19050ae640 10 monclient: get_auth_request con 0x557795e6ca40 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:55:19.865+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f800ebd0 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:55:19.866+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f1908027900 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:55:19.866+0000 7f19050ae640 10 monclient: get_auth_request con 0x7f1900053870 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:55:19.868+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f8036040 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:55:19.870+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f18f8018950 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:55:19.873+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557795e67bf0 msgr2=0x557795e69fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 1 client.6701.objecter ms_handle_reset 0x557795e67bf0 session 0x557795bd5320 osd.5 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 10 monclient: _renew_subs 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:55:19.874+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f18e8079e40 auth_method 0 2023-12-03T11:55:19.954 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zone default found 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Realm: () 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:19.955 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:55:19.876+0000 7f190fdaf800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zonegroup default 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started sync module instance, tier type = 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:55:19.882+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1908027900 msgr2=0x7f1908028410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:55:19.882+0000 7f18ee7fc640 1 client.6701.objecter ms_handle_reset 0x7f1908027900 session 0x7f1908021040 osd.2 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:55:19.882+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e67bf0 auth_method 0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=0 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=1 2023-12-03T11:55:19.956 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=2 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=3 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=4 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=5 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=6 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=7 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:55:19.885+0000 7f190fdaf800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.957 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:55:19.887+0000 7f18d37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache list hook 0x557795df2ef0 2023-12-03T11:55:19.958 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache inspect hook 0x557795df2ef0 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache erase hook 0x557795df2ef0 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache zap hook 0x557795df2ef0 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: get_monmap_and_config 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:55:19.888+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.960 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.961 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0af60e98 EEXIST 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795eaaa10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:19.962 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e797b0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e78bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.194632s 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e78bd0 auth_method 0 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795eaaa10 auth_method 0 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.963 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_done global_id 7107 payload 274 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient: _finish_hunting 0 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:55:19.889+0000 7f190e0b0640 1 monclient: found mon.noname-a 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:55:19.889+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e797b0 auth_method 0 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: dump: 2023-12-03T11:55:19.964 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: _finish_auth 0 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: _check_auth_tickets 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:55:19.889+0000 7f190fdaf800 5 monclient: authenticate success, global_id 7107 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.965 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: dump: 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:55:19.889+0000 7f190fdaf800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:55:19.889+0000 7f190fdaf800 4 set_mon_vals no callback set 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: shutdown 2023-12-03T11:55:19.966 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:19.967 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:19.968 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:55:19.890+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: starting msgr at 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: starting objecter 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command objecter_requests cmddesc objecter_requests hook 0x557795e78fb0 EEXIST 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: setting wanted keys 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: calling monclient init 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.969 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:55:19.891+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:55:19.891+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:55:19.891+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x557795eab920 EEXIST 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:55:19.970 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e78bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e797b0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795eaaa10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.197075s 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e78bd0 auth_method 0 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795eaaa10 auth_method 0 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.971 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e797b0 auth_method 0 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_done global_id 6707 payload 824 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient: _finish_hunting 0 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:55:19.891+0000 7f190e0b0640 1 monclient: found mon.noname-c 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.972 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: dump: 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.973 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: _finish_auth 0 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: _check_auth_tickets 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: dump: 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:19.974 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:55:19.892+0000 7f190fdaf800 5 monclient: authenticate success, global_id 6707 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:55:19.891+0000 7f18d2fe5640 4 set_mon_vals no callback set 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:55:19.892+0000 7f190fdaf800 1 librados: init done 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:55:19.892+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cr dump hook 0x557795eaa568 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.975 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 note: GC not initialized 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:55:19.893+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f1848069820 auth_method 0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:55:19.893+0000 7f18427fc640 20 reqs_thread_entry: start 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace show hook 0x7f19000043a0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace history hook 0x7f19000043a0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace active hook 0x7f19000043a0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace active_short hook 0x7f19000043a0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 Filter name: none 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:55:19.894+0000 7f1833fff640 20 reqs_thread_entry: start 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:19.976 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 : miss 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:55:19.895+0000 7f190fdaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:55:19.895+0000 7f19050ae640 10 monclient: get_auth_request con 0x557795ef8820 auth_method 0 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25, ofs0, ns] 2023-12-03T11:55:19.977 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660538 cookie 93972104423024 notifier 6701 bl.length()=893 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_sources= new_sources= 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential sources added=[] removed=[] 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_dests= new_dests= 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential dests added=[] removed=[] 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:19.978 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:19.979 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.0 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.2 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.3 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.4 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.5 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.6 2023-12-03T11:55:19.980 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.7 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.8 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.9 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.10 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f190fdaf800 time 2023-12-03T11:55:19.911578+0000 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f191476eef8] 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:19.981 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f190fdaf800 thread_name:radosgw-admin 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.982 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1912054db0] 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f19120a154c] 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f191476ef52] 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:19.983 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:19.984 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:55:19.985 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:55:19.986 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:55:19.987 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:55:19.988 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:55:19.989 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:55:19.990 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:55:19.991 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:55:19.992 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:55:19.993 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:55:19.994 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:55:19.995 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:55:19.996 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:55:19.997 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:55:19.998 INFO:tasks.workunit.client.0.smithi116.stderr: 7f1833fff640 / http_manager 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18427fc640 / http_manager 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d17e2640 / ms_dispatch 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d2fe5640 / io_context_pool 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d37e6640 / rgw_dt_lg_renew 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18ee7fc640 / ms_dispatch 2023-12-03T11:55:19.999 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18effff640 / io_context_pool 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: 7f19050ae640 / msgr-worker-2 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: 7f19077fe640 / ms_dispatch 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190d8af640 / msgr-worker-1 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190e0b0640 / msgr-worker-0 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190fdaf800 / radosgw-admin 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:55:20.000 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/log/ceph/ceph-client.admin.103693.log 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:20.001 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding auth protocol: cephx 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.002 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: crc 2023-12-03T11:55:20.003 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af607d0) adding con mode: secure 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:55:19.857+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:55:19.857+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:55:19.857+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0af60918 EEXIST 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:20.004 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e4c060 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795e47ff0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e4c440 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.005 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:55:19.857+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.163663s 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e47ff0 auth_method 0 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.006 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e4c060 auth_method 0 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.007 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e4c440 auth_method 0 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.008 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:55:19.858+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:55:19.858+0000 7f19077fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient(hunting): handle_auth_done global_id 7101 payload 274 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient: _finish_hunting 0 2023-12-03T11:55:20.009 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:55:19.858+0000 7f190d8af640 1 monclient: found mon.noname-a 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:55:19.858+0000 7f190d8af640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:55:19.858+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f0069820 auth_method 0 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: dump: 2023-12-03T11:55:20.010 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.011 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: _finish_auth 0 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: _check_auth_tickets 2023-12-03T11:55:20.012 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:55:19.858+0000 7f18effff640 10 monclient: dump: 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.014 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:55:19.858+0000 7f190fdaf800 5 monclient: authenticate success, global_id 7101 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:55:19.858+0000 7f190fdaf800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:55:19.858+0000 7f190fdaf800 4 set_mon_vals no callback set 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: shutdown 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:20.015 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:20.017 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding auth protocol: cephx 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.018 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: crc 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795e17610) adding con mode: secure 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:55:19.859+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.019 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: starting msgr at 2023-12-03T11:55:20.021 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: starting objecter 2023-12-03T11:55:20.021 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command objecter_requests cmddesc objecter_requests hook 0x557795e1bb80 EEXIST 2023-12-03T11:55:20.021 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: setting wanted keys 2023-12-03T11:55:20.021 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:55:19.859+0000 7f190fdaf800 1 librados: calling monclient init 2023-12-03T11:55:20.021 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:55:19.859+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding auth protocol: cephx 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.022 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: crc 2023-12-03T11:55:20.023 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:55:19.859+0000 7f190fdaf800 5 AuthRegistry(0x557795d520d8) adding con mode: secure 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:55:19.859+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:55:19.860+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:55:19.860+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x557795d52220 EEXIST 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e47ff0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795e4c060 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:20.024 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e55240 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.165801s 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e47ff0 auth_method 0 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e4c060 auth_method 0 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.025 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:55:19.860+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e55240 auth_method 0 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:55:19.860+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient(hunting): handle_auth_done global_id 6701 payload 824 2023-12-03T11:55:20.026 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient: _finish_hunting 0 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:55:19.860+0000 7f190d8af640 1 monclient: found mon.noname-c 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:55:19.860+0000 7f190d8af640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: dump: 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.027 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: _finish_auth 0 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: _check_auth_tickets 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:55:19.860+0000 7f18ee7fc640 10 monclient: dump: 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.028 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:55:19.860+0000 7f190fdaf800 5 monclient: authenticate success, global_id 6701 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:55:19.860+0000 7f18effff640 4 set_mon_vals no callback set 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:55:19.860+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:55:19.860+0000 7f190fdaf800 1 librados: init done 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:55:19.861+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:55:19.862+0000 7f18ee7fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:55:19.862+0000 7f19050ae640 10 monclient: get_auth_request con 0x7f18e8069850 auth_method 0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:55:19.862+0000 7f190e0b0640 10 monclient: get_auth_request con 0x557795e63970 auth_method 0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.029 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 realm 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:55:19.863+0000 7f190fdaf800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:55:19.863+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:55:19.863+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e67bf0 auth_method 0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:55:19.864+0000 7f190fdaf800 20 searching for the correct realm 2023-12-03T11:55:20.030 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:55:19.864+0000 7f19050ae640 10 monclient: get_auth_request con 0x557795e6ca40 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:55:19.865+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f800ebd0 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:55:19.866+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f1908027900 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:55:19.866+0000 7f19050ae640 10 monclient: get_auth_request con 0x7f1900053870 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:55:19.868+0000 7f190e0b0640 10 monclient: get_auth_request con 0x7f18f8036040 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:55:19.870+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f18f8018950 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:55:19.873+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x557795e67bf0 msgr2=0x557795e69fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 1 client.6701.objecter ms_handle_reset 0x557795e67bf0 session 0x557795bd5320 osd.5 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 10 monclient: _renew_subs 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:55:19.873+0000 7f18ee7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:55:19.874+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f18e8079e40 auth_method 0 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:20.031 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:20.032 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:55:19.875+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zone default found 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Realm: () 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:55:19.876+0000 7f190fdaf800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:55:19.876+0000 7f190fdaf800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:20.033 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 zonegroup default 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.034 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started sync module instance, tier type = 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:55:19.876+0000 7f190fdaf800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:55:19.882+0000 7f190d8af640 0 -- 172.21.15.116:0/399793749 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f1908027900 msgr2=0x7f1908028410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:55:19.882+0000 7f18ee7fc640 1 client.6701.objecter ms_handle_reset 0x7f1908027900 session 0x7f1908021040 osd.2 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:55:19.882+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e67bf0 auth_method 0 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=0 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=1 2023-12-03T11:55:20.035 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=2 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=3 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=4 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=5 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=6 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:55:19.885+0000 7f190fdaf800 20 add_watcher() i=7 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:55:19.885+0000 7f190fdaf800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:20.036 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding auth protocol: cephx 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:55:19.887+0000 7f18d37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.037 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: crc 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:55:19.887+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af61200) adding con mode: secure 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:55:19.888+0000 7f190fdaf800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache list hook 0x557795df2ef0 2023-12-03T11:55:20.038 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache inspect hook 0x557795df2ef0 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache erase hook 0x557795df2ef0 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cache zap hook 0x557795df2ef0 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: get_monmap_and_config 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:55:19.888+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:20.039 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding auth protocol: cephx 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.040 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x557795ea82c0) adding con mode: secure 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding auth protocol: cephx 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.041 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: crc 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 AuthRegistry(0x7ffd0af60d50) adding con mode: secure 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:55:19.888+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:55:19.888+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0af60e98 EEXIST 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795eaaa10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e797b0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e78bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.042 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:55:19.888+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.194632s 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795e78bd0 auth_method 0 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795eaaa10 auth_method 0 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:55:19.889+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.043 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient(hunting): handle_auth_done global_id 7107 payload 274 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient: _finish_hunting 0 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:55:19.889+0000 7f190e0b0640 1 monclient: found mon.noname-a 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:55:19.889+0000 7f190e0b0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:55:19.889+0000 7f190d8af640 10 monclient: get_auth_request con 0x557795e797b0 auth_method 0 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: dump: 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.044 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: _finish_auth 0 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: _check_auth_tickets 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:55:19.889+0000 7f190fdaf800 5 monclient: authenticate success, global_id 7107 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:55:19.889+0000 7f18d2fe5640 10 monclient: dump: 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.045 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:55:19.889+0000 7f190fdaf800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:55:19.889+0000 7f190fdaf800 4 set_mon_vals no callback set 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: shutdown 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: build_initial_monmap 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 build_initial for_mkfs: 0 2023-12-03T11:55:20.046 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: monmap: 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:20.047 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding auth protocol: cephx 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: crc 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795e9ea70) adding con mode: secure 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:55:19.890+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.048 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: starting msgr at 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: starting objecter 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command objecter_requests cmddesc objecter_requests hook 0x557795e78fb0 EEXIST 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: setting wanted keys 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:55:19.890+0000 7f190fdaf800 1 librados: calling monclient init 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:55:19.890+0000 7f190fdaf800 10 monclient: init 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding auth protocol: cephx 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.049 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: crc 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:55:19.890+0000 7f190fdaf800 5 AuthRegistry(0x557795eab7d8) adding con mode: secure 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:55:19.891+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:55:19.891+0000 7f190fdaf800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:55:19.891+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command rotate-key cmddesc rotate-key hook 0x557795eab920 EEXIST 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-c con 0x557795e78bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-b con 0x557795e797b0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): picked mon.noname-a con 0x557795eaaa10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.050 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:55:19.891+0000 7f190fdaf800 10 monclient(hunting): authenticate will time out at 1576.197075s 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): get_auth_request con 0x557795e78bd0 auth_method 0 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): get_auth_request con 0x557795eaaa10 auth_method 0 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:55:19.891+0000 7f19050ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:20.051 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): get_auth_request con 0x557795e797b0 auth_method 0 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:55:19.891+0000 7f190d8af640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient(hunting): handle_auth_done global_id 6707 payload 824 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient: _finish_hunting 0 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:55:19.891+0000 7f190e0b0640 1 monclient: found mon.noname-c 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:55:19.891+0000 7f190e0b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: dump: 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: _finish_auth 0 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: _check_auth_tickets 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:55:19.891+0000 7f18d17e2640 10 monclient: dump: 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:20.053 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:55:19.892+0000 7f190fdaf800 5 monclient: authenticate success, global_id 6707 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:55:19.891+0000 7f18d2fe5640 4 set_mon_vals no callback set 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _renew_subs 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:55:19.892+0000 7f190fdaf800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:55:19.892+0000 7f190fdaf800 1 librados: init done 2023-12-03T11:55:20.054 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:55:19.892+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command cr dump hook 0x557795eaa568 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:55:19.893+0000 7f18d17e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 note: GC not initialized 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:55:19.893+0000 7f190d8af640 10 monclient: get_auth_request con 0x7f1848069820 auth_method 0 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:55:19.893+0000 7f18427fc640 20 reqs_thread_entry: start 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace show hook 0x7f19000043a0 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace history hook 0x7f19000043a0 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace active hook 0x7f19000043a0 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:55:19.893+0000 7f190fdaf800 5 asok(0x557795c0ca10) register_command sync trace active_short hook 0x7f19000043a0 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 Filter name: none 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:55:19.894+0000 7f1833fff640 20 reqs_thread_entry: start 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:55:20.055 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 : miss 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:55:19.894+0000 7f190fdaf800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:55:19.894+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:55:19.895+0000 7f190fdaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:55:19.895+0000 7f19050ae640 10 monclient: get_auth_request con 0x557795ef8820 auth_method 0 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:20.056 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:55:19.900+0000 7f190fdaf800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25, ofs0, ns] 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660538 cookie 93972104423024 notifier 6701 bl.length()=893 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:55:19.901+0000 7f18effff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_sources= new_sources= 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential sources added=[] removed=[] 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_dests= new_dests= 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:55:19.901+0000 7f190fdaf800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential dests added=[] removed=[] 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:20.057 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:20.058 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:55:19.903+0000 7f190fdaf800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.0 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:55:19.903+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.2 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:55:19.904+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.3 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.4 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:55:19.905+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.5 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.6 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.7 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:55:19.906+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.8 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.9 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:55:19.907+0000 7f190fdaf800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.10 2023-12-03T11:55:20.059 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f190fdaf800 time 2023-12-03T11:55:19.911578+0000 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f191476eef8] 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:55:19.911+0000 7f190fdaf800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7f190fdaf800 thread_name:radosgw-admin 2023-12-03T11:55:20.060 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1912054db0] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f19120a154c] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f191476ef52] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f191476f0b6] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557793488a71] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x5577934951d8] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f191203feb0] 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:20.061 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:55:20.062 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:55:20.065 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:55:20.066 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 7f1833fff640 / http_manager 2023-12-03T11:55:20.067 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18427fc640 / http_manager 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d17e2640 / ms_dispatch 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d2fe5640 / io_context_pool 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18d37e6640 / rgw_dt_lg_renew 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18ee7fc640 / ms_dispatch 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f18effff640 / io_context_pool 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f19050ae640 / msgr-worker-2 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f19077fe640 / ms_dispatch 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190d8af640 / msgr-worker-1 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190e0b0640 / msgr-worker-0 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: 7f190fdaf800 / radosgw-admin 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:55:19.912917Z_4b55a49b-0c6d-4592-9b57-0db003240e3c/log 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:55:20.068 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.958+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.959+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.959+0000 7f5abfa16800 20 realm 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.959+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.960+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.960+0000 7f5abfa16800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.960+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.960+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.960+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.961+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.961+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.961+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.961+0000 7f5abfa16800 20 searching for the correct realm 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.970+0000 7f5abb093640 0 -- 172.21.15.116:0/525388992 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x559df1c9a410 msgr2=0x559df1c9c800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.972+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 zone default found 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 4 Realm: () 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 zonegroup default 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.973+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.974+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.974+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.974+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.974+0000 7f5abfa16800 20 started sync module instance, tier type = 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.974+0000 7f5abfa16800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.976+0000 7f5abb093640 0 -- 172.21.15.116:0/525388992 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x7f5ab005c5e0 msgr2=0x7f5a9407ef00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=0 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=1 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=2 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=3 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=4 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=5 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.982+0000 7f5abfa16800 20 add_watcher() i=6 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.983+0000 7f5abfa16800 20 add_watcher() i=7 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.983+0000 7f5abfa16800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.985+0000 7f5a817e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.985+0000 7f5abfa16800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.985+0000 7f5abfa16800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 5 note: GC not initialized 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f59effff640 20 reqs_thread_entry: start 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 20 Filter name: none 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f59edffb640 20 reqs_thread_entry: start 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.991+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:55:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.992+0000 7f5abfa16800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.993+0000 7f5abfa16800 20 rados->read ofs=0 len=0 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.994+0000 7f5abfa16800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.994+0000 7f5a817e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.996+0000 7f5abfa16800 20 remove_watcher() i=0 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.996+0000 7f5abfa16800 2 removed watcher, disabling cache 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.997+0000 7f5abfa16800 20 remove_watcher() i=1 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.998+0000 7f5abfa16800 20 remove_watcher() i=2 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:19.999+0000 7f5abfa16800 20 remove_watcher() i=3 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:20.000+0000 7f5abfa16800 20 remove_watcher() i=4 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:20.002+0000 7f5abfa16800 20 remove_watcher() i=5 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:20.003+0000 7f5abfa16800 20 remove_watcher() i=6 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:20.004+0000 7f5abfa16800 20 remove_watcher() i=7 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:20.079 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:20.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:20.080 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.050 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.093 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:55:50.127 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:50.127 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:50.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.141+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 realm 2023-12-03T11:55:50.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:50.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.143+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.144+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.144+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.144+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:50.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.144+0000 7fd492a8c800 20 searching for the correct realm 2023-12-03T11:55:50.156 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.155+0000 7fd487d8b640 0 -- 172.21.15.116:0/936762458 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55f6811550a0 msgr2=0x55f681157490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.156+0000 7fd492a8c800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:50.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.156+0000 7fd492a8c800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:50.158 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.156+0000 7fd492a8c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.156+0000 7fd492a8c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.156+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 zone default found 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 4 Realm: () 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 zonegroup default 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.157+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 started sync module instance, tier type = 2023-12-03T11:55:50.161 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.158+0000 7fd492a8c800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:50.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.164+0000 7fd490d8d640 0 -- 172.21.15.116:0/936762458 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7fd47803b040 msgr2=0x7fd47803e410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.169 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=0 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=1 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=2 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=3 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=4 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=5 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.168+0000 7fd492a8c800 20 add_watcher() i=6 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.169+0000 7fd492a8c800 20 add_watcher() i=7 2023-12-03T11:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.169+0000 7fd492a8c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.171+0000 7fd4567e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.171+0000 7fd492a8c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.171+0000 7fd492a8c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.177+0000 7fd492a8c800 5 note: GC not initialized 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.177+0000 7fd3c57fa640 20 reqs_thread_entry: start 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.177+0000 7fd492a8c800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.177+0000 7fd492a8c800 20 Filter name: none 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd3beffd640 20 reqs_thread_entry: start 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:55:50.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 : miss 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.178+0000 7fd492a8c800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.179+0000 7fd492a8c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:50.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.179+0000 7fd492a8c800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.179+0000 7fd492a8c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.179+0000 7fd492a8c800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.179+0000 7fd492a8c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 2023-12-03T11:55:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd492a8c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd492a8c800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd492a8c800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd492a8c800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25, ofs0, ns] 2023-12-03T11:55:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd486d89640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660540 cookie 94517215998656 notifier 7149 bl.length()=893 2023-12-03T11:55:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd486d89640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd486d89640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.184+0000 7fd486d89640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.185+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_sources= new_sources= 2023-12-03T11:55:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.185+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential sources added=[] removed=[] 2023-12-03T11:55:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.185+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_dests= new_dests= 2023-12-03T11:55:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.185+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential dests added=[] removed=[] 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.0 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.1 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.2 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.3 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.4 2023-12-03T11:55:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.5 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.6 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.7 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.8 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.9 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.10 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1.11 2023-12-03T11:55:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.187+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.0 2023-12-03T11:55:50.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.188+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.1 2023-12-03T11:55:50.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.188+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.2 2023-12-03T11:55:50.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.189+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.3 2023-12-03T11:55:50.191 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.189+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.4 2023-12-03T11:55:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.190+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.5 2023-12-03T11:55:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.190+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.6 2023-12-03T11:55:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.190+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.7 2023-12-03T11:55:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.191+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.8 2023-12-03T11:55:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.191+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.9 2023-12-03T11:55:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.192+0000 7fd492a8c800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.25.10 2023-12-03T11:55:50.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.195+0000 7fd492a8c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:50.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.195+0000 7fd492a8c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 2023-12-03T11:55:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd492a8c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd492a8c800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd492a8c800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd492a8c800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25, ofs0, ns] 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd486d89640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660541 cookie 94517215998656 notifier 7149 bl.length()=861 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd486d89640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 info.flags=0x17 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd486d89640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 to cache LRU end 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.196+0000 7fd486d89640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.197+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_sources= new_sources= 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.197+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential sources added=[] removed=[] 2023-12-03T11:55:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.197+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): orig_dests= new_dests= 2023-12-03T11:55:50.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.197+0000 7fd492a8c800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[21f2df23-11b5-4c91-821f-a8e55156195e.4668.25]): potential dests added=[] removed=[] 2023-12-03T11:55:50.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.199+0000 7fd492a8c800 1 execute INFO: reshard of bucket "abort-at-commit-target-layout" completed successfully 2023-12-03T11:55:50.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.201+0000 7fd4567e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.203+0000 7fd492a8c800 20 remove_watcher() i=0 2023-12-03T11:55:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.203+0000 7fd492a8c800 2 removed watcher, disabling cache 2023-12-03T11:55:50.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.204+0000 7fd492a8c800 20 remove_watcher() i=1 2023-12-03T11:55:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.205+0000 7fd492a8c800 20 remove_watcher() i=2 2023-12-03T11:55:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.206+0000 7fd492a8c800 20 remove_watcher() i=3 2023-12-03T11:55:50.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.208+0000 7fd492a8c800 20 remove_watcher() i=4 2023-12-03T11:55:50.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.209+0000 7fd492a8c800 20 remove_watcher() i=5 2023-12-03T11:55:50.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.210+0000 7fd492a8c800 20 remove_watcher() i=6 2023-12-03T11:55:50.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.211+0000 7fd492a8c800 20 remove_watcher() i=7 2023-12-03T11:55:50.218 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:55:50.245 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:50.245 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:50.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.257+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 realm 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.259+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.260+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.260+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.260+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.261+0000 7fe6b1d25800 20 searching for the correct realm 2023-12-03T11:55:50.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.270+0000 7fe6ad954640 0 -- 172.21.15.116:0/1834369072 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55b7c00ab990 msgr2=0x55b7c00add80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.271+0000 7fe6b1d25800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.271+0000 7fe6b1d25800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.271+0000 7fe6b1d25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.271+0000 7fe6b1d25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.271+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.274 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 zone default found 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 4 Realm: () 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 zonegroup default 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.275 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.272+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.273+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.273+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.273+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.273+0000 7fe6b1d25800 20 started sync module instance, tier type = 2023-12-03T11:55:50.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.273+0000 7fe6b1d25800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.278+0000 7fe6ad954640 0 -- 172.21.15.116:0/1834369072 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7fe6a800e070 msgr2=0x7fe6a8024420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=0 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=1 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=2 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=3 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=4 2023-12-03T11:55:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=5 2023-12-03T11:55:50.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.282+0000 7fe6b1d25800 20 add_watcher() i=6 2023-12-03T11:55:50.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.283+0000 7fe6b1d25800 20 add_watcher() i=7 2023-12-03T11:55:50.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.283+0000 7fe6b1d25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:50.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.284+0000 7fe6737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.285+0000 7fe6b1d25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.285+0000 7fe6b1d25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.290+0000 7fe6b1d25800 5 note: GC not initialized 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.290+0000 7fe5e27fc640 20 reqs_thread_entry: start 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe6b1d25800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe6b1d25800 20 Filter name: none 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe5d3fff640 20 reqs_thread_entry: start 2023-12-03T11:55:50.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:50.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.291+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.292+0000 7fe6b1d25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:50.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.293+0000 7fe6b1d25800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.293+0000 7fe6b1d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.294+0000 7fe6737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.296+0000 7fe6b1d25800 20 remove_watcher() i=0 2023-12-03T11:55:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.296+0000 7fe6b1d25800 2 removed watcher, disabling cache 2023-12-03T11:55:50.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.297+0000 7fe6b1d25800 20 remove_watcher() i=1 2023-12-03T11:55:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.298+0000 7fe6b1d25800 20 remove_watcher() i=2 2023-12-03T11:55:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.299+0000 7fe6b1d25800 20 remove_watcher() i=3 2023-12-03T11:55:50.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.300+0000 7fe6b1d25800 20 remove_watcher() i=4 2023-12-03T11:55:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.301+0000 7fe6b1d25800 20 remove_watcher() i=5 2023-12-03T11:55:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.302+0000 7fe6b1d25800 20 remove_watcher() i=6 2023-12-03T11:55:50.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.303+0000 7fe6b1d25800 20 remove_watcher() i=7 2023-12-03T11:55:50.309 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:50.309 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:50.309 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-commit-target-layout id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.25 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:55:50.309 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:50.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.311 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.315 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.358 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.383 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with EIO injected at do_reshard 2023-12-03T11:55:50.384 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.385 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.396 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.439 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.447 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.455 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.463 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.471 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.478 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.486 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.495 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.502 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.509 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.518 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.525 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.532 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.540 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.547 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.555 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.562 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.571 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:50.586 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:50.593 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:55:50.620 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:50.621 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:50.634 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.633+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 realm 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.634+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.635+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.635+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.635+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:50.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.635+0000 7f08a211e800 20 searching for the correct realm 2023-12-03T11:55:50.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.645+0000 7f089f424640 0 -- 172.21.15.116:0/657399777 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55da16c37f30 msgr2=0x55da16c3a320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.646+0000 7f08a211e800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:50.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.646+0000 7f08a211e800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:50.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.646+0000 7f08a211e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:50.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.646+0000 7f08a211e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:50.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.646+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:50.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 zone default found 2023-12-03T11:55:50.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 4 Realm: () 2023-12-03T11:55:50.650 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 zonegroup default 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.647+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.648+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.648+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.648+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.648+0000 7f08a211e800 20 started sync module instance, tier type = 2023-12-03T11:55:50.651 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.648+0000 7f08a211e800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:50.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.652+0000 7f089fc25640 0 -- 172.21.15.116:0/657399777 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f0898023040 msgr2=0x7f0898023420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=0 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=1 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=2 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=3 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=4 2023-12-03T11:55:50.659 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=5 2023-12-03T11:55:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.658+0000 7f08a211e800 20 add_watcher() i=6 2023-12-03T11:55:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.659+0000 7f08a211e800 20 add_watcher() i=7 2023-12-03T11:55:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.659+0000 7f08a211e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:50.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.660+0000 7f0865fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.661+0000 7f08a211e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:50.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.661+0000 7f08a211e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:50.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.666+0000 7f08a211e800 5 note: GC not initialized 2023-12-03T11:55:50.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.667+0000 7f07dcff9640 20 reqs_thread_entry: start 2023-12-03T11:55:50.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.667+0000 7f08a211e800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:50.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.667+0000 7f08a211e800 20 Filter name: none 2023-12-03T11:55:50.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.667+0000 7f07d27fc640 20 reqs_thread_entry: start 2023-12-03T11:55:50.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.667+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.668+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.668+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.668+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.668+0000 7f08a211e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.668+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.669+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:50.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.669+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.669+0000 7f08a211e800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:50.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.669+0000 7f08a211e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:50.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.670+0000 7f08a211e800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.670+0000 7f08a211e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.671+0000 7f0865fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.673+0000 7f08a211e800 20 remove_watcher() i=0 2023-12-03T11:55:50.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.673+0000 7f08a211e800 2 removed watcher, disabling cache 2023-12-03T11:55:50.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.674+0000 7f08a211e800 20 remove_watcher() i=1 2023-12-03T11:55:50.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.675+0000 7f08a211e800 20 remove_watcher() i=2 2023-12-03T11:55:50.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.675+0000 7f08a211e800 20 remove_watcher() i=3 2023-12-03T11:55:50.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.676+0000 7f08a211e800 20 remove_watcher() i=4 2023-12-03T11:55:50.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.677+0000 7f08a211e800 20 remove_watcher() i=5 2023-12-03T11:55:50.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.678+0000 7f08a211e800 20 remove_watcher() i=6 2023-12-03T11:55:50.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.679+0000 7f08a211e800 20 remove_watcher() i=7 2023-12-03T11:55:50.685 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:50.685 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:50.685 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:50.685 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:50.686 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-error-at do_reshard 2023-12-03T11:55:50.718 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:50.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.730+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 20 realm 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.932+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.933+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.933+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.934+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.934+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.934+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.934+0000 7f9312a11800 20 searching for the correct realm 2023-12-03T11:55:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.942+0000 7f9310d12640 0 -- 172.21.15.116:0/3414157928 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x5625a682d000 msgr2=0x5625a682f3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.944+0000 7f9312a11800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.944+0000 7f9312a11800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.944+0000 7f9312a11800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.944+0000 7f9312a11800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.944+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 zone default found 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 4 Realm: () 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 zonegroup default 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.945+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.946+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.946+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.946+0000 7f9312a11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:50.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.946+0000 7f9312a11800 20 started sync module instance, tier type = 2023-12-03T11:55:50.949 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.946+0000 7f9312a11800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.949+0000 7f9310d12640 0 -- 172.21.15.116:0/3414157928 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f92f800e050 msgr2=0x7f92f8019910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=0 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=1 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=2 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=3 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=4 2023-12-03T11:55:50.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=5 2023-12-03T11:55:50.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.955+0000 7f9312a11800 20 add_watcher() i=6 2023-12-03T11:55:50.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.956+0000 7f9312a11800 20 add_watcher() i=7 2023-12-03T11:55:50.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.956+0000 7f9312a11800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:50.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.957+0000 7f92d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.958+0000 7f9312a11800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:50.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.958+0000 7f9312a11800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:50.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.963+0000 7f9312a11800 5 note: GC not initialized 2023-12-03T11:55:50.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f92457fa640 20 reqs_thread_entry: start 2023-12-03T11:55:50.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:50.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 20 Filter name: none 2023-12-03T11:55:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f923effd640 20 reqs_thread_entry: start 2023-12-03T11:55:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2023-12-03T11:55:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 : miss 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.964+0000 7f9312a11800 20 rados->read ofs=0 len=0 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.965+0000 7f9312a11800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 2023-12-03T11:55:50.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:50.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:50.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26, ofs0, ns] 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9306d0e640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660567 cookie 94719707375008 notifier 6794 bl.length()=873 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9306d0e640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9306d0e640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9306d0e640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_sources= new_sources= 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential sources added=[] removed=[] 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_dests= new_dests= 2023-12-03T11:55:50.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.970+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential dests added=[] removed=[] 2023-12-03T11:55:50.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.973+0000 7f9312a11800 -1 Injecting error=-5 at location=do_reshard 2023-12-03T11:55:50.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.977+0000 7f9312a11800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:50.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.977+0000 7f9312a11800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.978+0000 7f9312a11800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.978+0000 7f9312a11800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.978+0000 7f9312a11800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.978+0000 7f9312a11800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26, ofs0, ns] 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.978+0000 7f9306d0e640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660568 cookie 94719707375008 notifier 6794 bl.length()=841 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9306d0e640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9306d0e640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9306d0e640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:50.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_sources= new_sources= 2023-12-03T11:55:50.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential sources added=[] removed=[] 2023-12-03T11:55:50.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_dests= new_dests= 2023-12-03T11:55:50.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9312a11800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential dests added=[] removed=[] 2023-12-03T11:55:50.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.979+0000 7f9312a11800 1 execute INFO: reshard of bucket "error-at-do-reshard" canceled due to errors 2023-12-03T11:55:50.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.981+0000 7f92d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:50.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.983+0000 7f9312a11800 20 remove_watcher() i=0 2023-12-03T11:55:50.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.983+0000 7f9312a11800 2 removed watcher, disabling cache 2023-12-03T11:55:50.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.984+0000 7f9312a11800 20 remove_watcher() i=1 2023-12-03T11:55:50.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.985+0000 7f9312a11800 20 remove_watcher() i=2 2023-12-03T11:55:50.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.986+0000 7f9312a11800 20 remove_watcher() i=3 2023-12-03T11:55:50.988 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.987+0000 7f9312a11800 20 remove_watcher() i=4 2023-12-03T11:55:50.989 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.988+0000 7f9312a11800 20 remove_watcher() i=5 2023-12-03T11:55:50.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.988+0000 7f9312a11800 20 remove_watcher() i=6 2023-12-03T11:55:50.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:50.989+0000 7f9312a11800 20 remove_watcher() i=7 2023-12-03T11:55:50.997 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:55:51.024 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.024 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.036+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 realm 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.040 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.038+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.039+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.039+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.039+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.039+0000 7f0189f23800 20 searching for the correct realm 2023-12-03T11:55:51.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f018533b640 0 -- 172.21.15.116:0/1080504605 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5581d179a2a0 msgr2=0x5581d179c690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.049+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 zone default found 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 4 Realm: () 2023-12-03T11:55:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 zonegroup default 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.050+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.051+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.051+0000 7f0189f23800 20 started sync module instance, tier type = 2023-12-03T11:55:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.051+0000 7f0189f23800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.057+0000 7f018633d640 0 -- 172.21.15.116:0/1080504605 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f017c056ab0 msgr2=0x7f017c0899e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=0 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=1 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=2 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=3 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=4 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=5 2023-12-03T11:55:51.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.059+0000 7f0189f23800 20 add_watcher() i=6 2023-12-03T11:55:51.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.060+0000 7f0189f23800 20 add_watcher() i=7 2023-12-03T11:55:51.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.060+0000 7f0189f23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.061+0000 7f014b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.062+0000 7f0189f23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.062+0000 7f0189f23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.067+0000 7f0189f23800 5 note: GC not initialized 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.067+0000 7f00ba7fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.067+0000 7f0189f23800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.068+0000 7f0189f23800 20 Filter name: none 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.068+0000 7f00abfff640 20 reqs_thread_entry: start 2023-12-03T11:55:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.068+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.068+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.068+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.069+0000 7f0189f23800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.071+0000 7f0189f23800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.071+0000 7f0189f23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.072+0000 7f014b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.073+0000 7f0189f23800 20 remove_watcher() i=0 2023-12-03T11:55:51.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.073+0000 7f0189f23800 2 removed watcher, disabling cache 2023-12-03T11:55:51.077 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.075+0000 7f0189f23800 20 remove_watcher() i=1 2023-12-03T11:55:51.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.076+0000 7f0189f23800 20 remove_watcher() i=2 2023-12-03T11:55:51.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.077+0000 7f0189f23800 20 remove_watcher() i=3 2023-12-03T11:55:51.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.078+0000 7f0189f23800 20 remove_watcher() i=4 2023-12-03T11:55:51.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.079+0000 7f0189f23800 20 remove_watcher() i=5 2023-12-03T11:55:51.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.080+0000 7f0189f23800 20 remove_watcher() i=6 2023-12-03T11:55:51.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.081+0000 7f0189f23800 20 remove_watcher() i=7 2023-12-03T11:55:51.087 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.087 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:51.087 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:51.087 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.089 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.096 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.139 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:55:51.168 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.168 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.182 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.180+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 20 realm 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.182+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.183+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.183+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.183+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.183+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.183+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.184+0000 7f7e4b8ba800 20 searching for the correct realm 2023-12-03T11:55:51.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.193+0000 7f7e493ba640 0 -- 172.21.15.116:0/1011073273 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5577375e15a0 msgr2=0x5577375e3990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.196 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.194+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 zone default found 2023-12-03T11:55:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 4 Realm: () 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 zonegroup default 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.195+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.196+0000 7f7e4b8ba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.196+0000 7f7e4b8ba800 20 started sync module instance, tier type = 2023-12-03T11:55:51.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.196+0000 7f7e4b8ba800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.203+0000 7f7e49bbb640 0 -- 172.21.15.116:0/1011073273 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f7e34066070 msgr2=0x7f7e3405d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=0 2023-12-03T11:55:51.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=1 2023-12-03T11:55:51.206 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=2 2023-12-03T11:55:51.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=3 2023-12-03T11:55:51.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=4 2023-12-03T11:55:51.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=5 2023-12-03T11:55:51.207 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.205+0000 7f7e4b8ba800 20 add_watcher() i=6 2023-12-03T11:55:51.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.207+0000 7f7e4b8ba800 20 add_watcher() i=7 2023-12-03T11:55:51.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.207+0000 7f7e4b8ba800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.208+0000 7f7e0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.208+0000 7f7e4b8ba800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.210 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.208+0000 7f7e4b8ba800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.214+0000 7f7e4b8ba800 5 note: GC not initialized 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.214+0000 7f7d7dffb640 20 reqs_thread_entry: start 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 20 Filter name: none 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7d6f7fe640 20 reqs_thread_entry: start 2023-12-03T11:55:51.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2023-12-03T11:55:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2023-12-03T11:55:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 : miss 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.215+0000 7f7e4b8ba800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.216+0000 7f7e4b8ba800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e4b8ba800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e4b8ba800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e4b8ba800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e4b8ba800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26, ofs0, ns] 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e2b7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660569 cookie 93970518387472 notifier 7257 bl.length()=873 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e2b7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e2b7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.221+0000 7f7e2b7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.222+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_sources= new_sources= 2023-12-03T11:55:51.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.222+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential sources added=[] removed=[] 2023-12-03T11:55:51.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.222+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_dests= new_dests= 2023-12-03T11:55:51.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.222+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential dests added=[] removed=[] 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.0 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.0 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.1 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.1 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.2 2023-12-03T11:55:51.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.2 2023-12-03T11:55:51.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.3 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.3 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.4 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.4 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.5 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.5 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.6 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.6 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.7 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.7 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.8 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.8 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.9 2023-12-03T11:55:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.9 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.10 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.10 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.11 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1.11 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.224+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.0 2023-12-03T11:55:51.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.225+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.1 2023-12-03T11:55:51.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.225+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.2 2023-12-03T11:55:51.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.226+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.3 2023-12-03T11:55:51.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.226+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.4 2023-12-03T11:55:51.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.226+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.5 2023-12-03T11:55:51.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.227+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.6 2023-12-03T11:55:51.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.228+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.7 2023-12-03T11:55:51.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.228+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.8 2023-12-03T11:55:51.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.228+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.9 2023-12-03T11:55:51.231 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.229+0000 7f7e4b8ba800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.26.10 2023-12-03T11:55:51.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.232+0000 7f7e4b8ba800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 : hit (requested=0x17, cached=0x17) 2023-12-03T11:55:51.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.232+0000 7f7e4b8ba800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 2023-12-03T11:55:51.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:51.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:51.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.234 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26, ofs0, ns] 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e2affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660570 cookie 93970518387472 notifier 7257 bl.length()=841 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e2affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 info.flags=0x17 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e2affd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 to cache LRU end 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e2affd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_sources= new_sources= 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential sources added=[] removed=[] 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): orig_dests= new_dests= 2023-12-03T11:55:51.235 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.233+0000 7f7e4b8ba800 20 handle_bi_update(): bucket=:error-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.26]): potential dests added=[] removed=[] 2023-12-03T11:55:51.237 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.236+0000 7f7e4b8ba800 1 execute INFO: reshard of bucket "error-at-do-reshard" completed successfully 2023-12-03T11:55:51.238 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.237+0000 7f7e0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.240 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.239+0000 7f7e4b8ba800 20 remove_watcher() i=0 2023-12-03T11:55:51.241 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.239+0000 7f7e4b8ba800 2 removed watcher, disabling cache 2023-12-03T11:55:51.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.240+0000 7f7e4b8ba800 20 remove_watcher() i=1 2023-12-03T11:55:51.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.241+0000 7f7e4b8ba800 20 remove_watcher() i=2 2023-12-03T11:55:51.243 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.242+0000 7f7e4b8ba800 20 remove_watcher() i=3 2023-12-03T11:55:51.244 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.243+0000 7f7e4b8ba800 20 remove_watcher() i=4 2023-12-03T11:55:51.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.244+0000 7f7e4b8ba800 20 remove_watcher() i=5 2023-12-03T11:55:51.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.245+0000 7f7e4b8ba800 20 remove_watcher() i=6 2023-12-03T11:55:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.245+0000 7f7e4b8ba800 20 remove_watcher() i=7 2023-12-03T11:55:51.253 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:55:51.280 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.280 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.292+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 20 realm 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.293+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.294+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.294+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.294+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.294+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.295+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.295+0000 7fc8a9410800 20 searching for the correct realm 2023-12-03T11:55:51.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.304+0000 7fc8a4b3c640 0 -- 172.21.15.116:0/372707440 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55b854b14f90 msgr2=0x55b854b173a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.305+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 zone default found 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 4 Realm: () 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 zonegroup default 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 started sync module instance, tier type = 2023-12-03T11:55:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.306+0000 7fc8a9410800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.314+0000 7fc8a433b640 0 -- 172.21.15.116:0/372707440 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55b854b19fe0 msgr2=0x55b854b1c3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=0 2023-12-03T11:55:51.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=1 2023-12-03T11:55:51.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=2 2023-12-03T11:55:51.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=3 2023-12-03T11:55:51.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=4 2023-12-03T11:55:51.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=5 2023-12-03T11:55:51.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.315+0000 7fc8a9410800 20 add_watcher() i=6 2023-12-03T11:55:51.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.316+0000 7fc8a9410800 20 add_watcher() i=7 2023-12-03T11:55:51.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.316+0000 7fc8a9410800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.317+0000 7fc86afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.317+0000 7fc8a9410800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.317+0000 7fc8a9410800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.323+0000 7fc8a9410800 5 note: GC not initialized 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.323+0000 7fc7d9ffb640 20 reqs_thread_entry: start 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.323+0000 7fc8a9410800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.323+0000 7fc8a9410800 20 Filter name: none 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.324+0000 7fc7cb7fe640 20 reqs_thread_entry: start 2023-12-03T11:55:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.324+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.324+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.324+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.327 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.325+0000 7fc8a9410800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.326+0000 7fc8a9410800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.327+0000 7fc8a9410800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.327+0000 7fc86afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.329+0000 7fc8a9410800 20 remove_watcher() i=0 2023-12-03T11:55:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.329+0000 7fc8a9410800 2 removed watcher, disabling cache 2023-12-03T11:55:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.330+0000 7fc8a9410800 20 remove_watcher() i=1 2023-12-03T11:55:51.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.331+0000 7fc8a9410800 20 remove_watcher() i=2 2023-12-03T11:55:51.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.332+0000 7fc8a9410800 20 remove_watcher() i=3 2023-12-03T11:55:51.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.333+0000 7fc8a9410800 20 remove_watcher() i=4 2023-12-03T11:55:51.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.334+0000 7fc8a9410800 20 remove_watcher() i=5 2023-12-03T11:55:51.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.335+0000 7fc8a9410800 20 remove_watcher() i=6 2023-12-03T11:55:51.337 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.336+0000 7fc8a9410800 20 remove_watcher() i=7 2023-12-03T11:55:51.342 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.342 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:51.342 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: error-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.26 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:55:51.342 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.344 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.347 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.409 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.434 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: reshard bucket with abort at do_reshard 2023-12-03T11:55:51.435 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.436 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.449 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.492 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.499 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.507 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.514 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.522 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.530 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.538 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.545 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.552 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.560 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.568 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.575 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.582 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.590 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.601 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.606 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.614 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.623 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.630 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.638 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:55:51.647 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:55:51.673 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.673 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.686+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 realm 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.688+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.689+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.689+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.689+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.691 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.689+0000 7f8163f40800 20 searching for the correct realm 2023-12-03T11:55:51.698 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.696+0000 7f815fd03640 0 -- 172.21.15.116:0/4013843877 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f81480a14a0 msgr2=0x7f8148099b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.700+0000 7f815fd03640 0 -- 172.21.15.116:0/4013843877 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55c8c63fa610 msgr2=0x55c8c63fca00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 20 zone default found 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 4 Realm: () 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.701+0000 7f8163f40800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 zonegroup default 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.704 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 started sync module instance, tier type = 2023-12-03T11:55:51.705 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.702+0000 7f8163f40800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=0 2023-12-03T11:55:51.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=1 2023-12-03T11:55:51.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=2 2023-12-03T11:55:51.712 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=3 2023-12-03T11:55:51.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=4 2023-12-03T11:55:51.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=5 2023-12-03T11:55:51.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.711+0000 7f8163f40800 20 add_watcher() i=6 2023-12-03T11:55:51.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.712+0000 7f8163f40800 20 add_watcher() i=7 2023-12-03T11:55:51.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.712+0000 7f8163f40800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.713+0000 7f8125fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.713+0000 7f8163f40800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.713+0000 7f8163f40800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f8163f40800 5 note: GC not initialized 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f809cff9640 20 reqs_thread_entry: start 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f8163f40800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f8163f40800 20 Filter name: none 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f80927fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.719+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.720+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.720+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.722 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.723 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.721+0000 7f8163f40800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.722+0000 7f8163f40800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.722+0000 7f8163f40800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.724 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.723+0000 7f8125fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.725+0000 7f8163f40800 20 remove_watcher() i=0 2023-12-03T11:55:51.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.725+0000 7f8163f40800 2 removed watcher, disabling cache 2023-12-03T11:55:51.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.726+0000 7f8163f40800 20 remove_watcher() i=1 2023-12-03T11:55:51.729 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.727+0000 7f8163f40800 20 remove_watcher() i=2 2023-12-03T11:55:51.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.728+0000 7f8163f40800 20 remove_watcher() i=3 2023-12-03T11:55:51.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.729+0000 7f8163f40800 20 remove_watcher() i=4 2023-12-03T11:55:51.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.730+0000 7f8163f40800 20 remove_watcher() i=5 2023-12-03T11:55:51.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.731+0000 7f8163f40800 20 remove_watcher() i=6 2023-12-03T11:55:51.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.732+0000 7f8163f40800 20 remove_watcher() i=7 2023-12-03T11:55:51.738 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.738 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:51.738 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:51.738 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.739 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 12 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at do_reshard 2023-12-03T11:55:51.771 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.771 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.783+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 realm 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.786+0000 7fb159fdc800 20 searching for the correct realm 2023-12-03T11:55:51.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.795+0000 7fb157adc640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55cba7b0a910 msgr2=0x55cba7b0cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zone default found 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Realm: () 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zonegroup default 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started sync module instance, tier type = 2023-12-03T11:55:51.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.803+0000 7fb1582dd640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fb1400573c0 msgr2=0x7fb1400769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=0 2023-12-03T11:55:51.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=1 2023-12-03T11:55:51.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=2 2023-12-03T11:55:51.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=3 2023-12-03T11:55:51.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=4 2023-12-03T11:55:51.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=5 2023-12-03T11:55:51.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=6 2023-12-03T11:55:51.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.808+0000 7fb159fdc800 20 add_watcher() i=7 2023-12-03T11:55:51.809 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.808+0000 7fb159fdc800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.809+0000 7fb11dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb159fdc800 5 note: GC not initialized 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb094ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb159fdc800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb159fdc800 20 Filter name: none 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb08a7fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:55:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.815+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 : miss 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.816+0000 7fb159fdc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 2023-12-03T11:55:51.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.821+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.821+0000 7fb159fdc800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.821+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.821+0000 7fb159fdc800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27, ofs0, ns] 2023-12-03T11:55:51.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660546 cookie 140399528550400 notifier 6896 bl.length()=873 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_sources= new_sources= 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential sources added=[] removed=[] 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_dests= new_dests= 2023-12-03T11:55:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential dests added=[] removed=[] 2023-12-03T11:55:51.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fb159fdc800 time 2023-12-03T11:55:51.825730+0000 2023-12-03T11:55:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:51.826 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.826 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fb15e96eef8] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.825+0000 7fb159fdc800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fb159fdc800 time 2023-12-03T11:55:51.825730+0000 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fb15e96eef8] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.827 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:55:51.828 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7fb159fdc800 thread_name:radosgw-admin 2023-12-03T11:55:51.846 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb15c254db0] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fb15c2a154c] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb15e96ef52] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.826+0000 7fb159fdc800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7fb159fdc800 thread_name:radosgw-admin 2023-12-03T11:55:51.847 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb15c254db0] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fb15c2a154c] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb15e96ef52] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.848 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:55:51.778+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: get_monmap_and_config 2023-12-03T11:55:51.849 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:55:51.778+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.850 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.851 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.852 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2241ad68 EEXIST 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7aee350 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7aee730 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7ae8a10 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.085546s 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7aee350 auth_method 0 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.853 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7aee730 auth_method 0 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7ae8a10 auth_method 0 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.854 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_done global_id 7320 payload 274 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:55:51.780+0000 7fb1582dd640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: dump: 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.855 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: dump: 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:55:51.780+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7320 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:55:51.780+0000 7fb159fdc800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:55:51.780+0000 7fb159fdc800 4 set_mon_vals no callback set 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb138069820 auth_method 0 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:55:51.780+0000 7fb159fdc800 10 monclient: shutdown 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.857 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.858 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: starting msgr at 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: starting objecter 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command objecter_requests cmddesc objecter_requests hook 0x55cba7ae9f00 EEXIST 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: setting wanted keys 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: calling monclient init 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.859 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.860 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x55cba7ab8300 EEXIST 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7ae8a10 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7aee350 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7ae6830 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.087692s 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7aee350 auth_method 0 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.861 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7ae6830 auth_method 0 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7ae8a10 auth_method 0 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.862 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_done global_id 6896 payload 824 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:55:51.782+0000 7fb14f2db640 1 monclient: found mon.noname-c 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: dump: 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.863 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: dump: 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.864 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:55:51.782+0000 7fb159fdc800 5 monclient: authenticate success, global_id 6896 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:55:51.782+0000 7fb14e2d9640 4 set_mon_vals no callback set 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:55:51.782+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:55:51.783+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.865 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:55:51.784+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb130069820 auth_method 0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:55:51.784+0000 7fb1582dd640 10 monclient: get_auth_request con 0x55cba7b06b80 auth_method 0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 realm 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:55:51.785+0000 7fb159fdc800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:55:51.786+0000 7fb157adc640 10 monclient: get_auth_request con 0x55cba7b0a910 auth_method 0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.866 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 searching for the correct realm 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:55:51.786+0000 7fb14f2db640 10 monclient: get_auth_request con 0x55cba7b0f850 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:55:51.787+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb140058130 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:55:51.788+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb15007b980 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:55:51.788+0000 7fb14f2db640 10 monclient: get_auth_request con 0x7fb14805e040 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:55:51.790+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb1400573c0 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:55:51.792+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb14005fc90 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:55:51.795+0000 7fb157adc640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55cba7b0a910 msgr2=0x55cba7b0cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:55:51.795+0000 7fb14cad6640 1 client.6896.objecter ms_handle_reset 0x55cba7b0a910 session 0x55cba7877320 osd.5 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:55:51.795+0000 7fb14cad6640 10 monclient: _renew_subs 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:55:51.795+0000 7fb14cad6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:55:51.795+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb130079e10 auth_method 0 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.867 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zone default found 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Realm: () 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:55:51.798+0000 7fb159fdc800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zonegroup default 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.868 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started sync module instance, tier type = 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:55:51.803+0000 7fb1582dd640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fb1400573c0 msgr2=0x7fb1400769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:55:51.803+0000 7fb14cad6640 1 client.6896.objecter ms_handle_reset 0x7fb1400573c0 session 0x7fb140060120 osd.6 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:55:51.804+0000 7fb1582dd640 10 monclient: get_auth_request con 0x55cba7b0a910 auth_method 0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=0 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=1 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=2 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=3 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=4 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=5 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=6 2023-12-03T11:55:51.869 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:55:51.808+0000 7fb159fdc800 20 add_watcher() i=7 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:55:51.808+0000 7fb159fdc800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.870 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:55:51.809+0000 7fb11dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:55:51.809+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache list hook 0x55cba7a94700 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache inspect hook 0x55cba7a94700 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache erase hook 0x55cba7a94700 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache zap hook 0x55cba7a94700 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: get_monmap_and_config 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:55:51.809+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.872 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.873 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2241b2e8 EEXIST 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7b49770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7b474c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.874 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7b40610 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.116432s 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7b49770 auth_method 0 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7b474c0 auth_method 0 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.875 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:55:51.810+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7b40610 auth_method 0 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient(hunting): handle_auth_done global_id 7326 payload 274 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:55:51.811+0000 7fb157adc640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.876 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: dump: 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.877 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: dump: 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:55:51.811+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7326 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:55:51.811+0000 7fb159fdc800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:51.878 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:55:51.811+0000 7fb159fdc800 4 set_mon_vals no callback set 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: shutdown 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.879 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:55:51.812+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.880 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: starting msgr at 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: starting objecter 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command objecter_requests cmddesc objecter_requests hook 0x55cba7b14560 EEXIST 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: setting wanted keys 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: calling monclient init 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.881 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:55:51.812+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:55:51.813+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x55cba7b4e690 EEXIST 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7b40610 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7b474c0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7b49770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.882 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.118842s 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7b40610 auth_method 0 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7b474c0 auth_method 0 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7b49770 auth_method 0 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.883 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_done global_id 7329 payload 824 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:55:51.813+0000 7fb1582dd640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: dump: 2023-12-03T11:55:51.884 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: dump: 2023-12-03T11:55:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7329 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:55:51.813+0000 7fb11d7e2640 4 set_mon_vals no callback set 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.886 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:55:51.813+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cr dump hook 0x55cba7b4fd28 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 note: GC not initialized 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:55:51.815+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb08c069820 auth_method 0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:55:51.815+0000 7fb094ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace show hook 0x7fb1500047e0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace history hook 0x7fb1500047e0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace active hook 0x7fb1500047e0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace active_short hook 0x7fb1500047e0 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 Filter name: none 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:55:51.815+0000 7fb08a7fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.887 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:55:51.815+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 : miss 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:55:51.816+0000 7fb14f2db640 10 monclient: get_auth_request con 0x55cba7b9b1c0 auth_method 0 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.888 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27, ofs0, ns] 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660546 cookie 140399528550400 notifier 6896 bl.length()=873 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_sources= new_sources= 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential sources added=[] removed=[] 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_dests= new_dests= 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential dests added=[] removed=[] 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:55:51.825+0000 7fb159fdc800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fb159fdc800 time 2023-12-03T11:55:51.825730+0000 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.889 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fb15e96eef8] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:55:51.826+0000 7fb159fdc800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7fb159fdc800 thread_name:radosgw-admin 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb15c254db0] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fb15c2a154c] 2023-12-03T11:55:51.890 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb15e96ef52] 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:55:51.891 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:55:51.892 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:55:51.893 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:55:51.894 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:55:51.895 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:55:51.896 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb08a7fc640 / http_manager 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb094ff9640 / http_manager 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb0977fe640 / ms_dispatch 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb11d7e2640 / io_context_pool 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb11dfe3640 / rgw_dt_lg_renew 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14cad6640 / ms_dispatch 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14dad8640 / io_context_pool 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14e2d9640 / io_context_pool 2023-12-03T11:55:51.897 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14f2db640 / msgr-worker-2 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb155ad8640 / ms_dispatch 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb157adc640 / msgr-worker-1 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb1582dd640 / msgr-worker-0 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb159fdc800 / radosgw-admin 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/log/ceph/ceph-client.admin.104513.log 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr:--- begin dump of recent events --- 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -499> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -498> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -497> 2023-12-03T11:55:51.778+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -496> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: get_monmap_and_config 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -495> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.898 INFO:tasks.workunit.client.0.smithi116.stderr: -494> 2023-12-03T11:55:51.778+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -493> 2023-12-03T11:55:51.778+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -492> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -491> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -490> 2023-12-03T11:55:51.778+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding auth protocol: cephx 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -489> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -488> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.899 INFO:tasks.workunit.client.0.smithi116.stderr: -487> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -486> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -485> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -484> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -483> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -482> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -481> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -480> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -479> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: crc 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -478> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9ab0) adding con mode: secure 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -477> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -476> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -475> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -474> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -473> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding auth protocol: cephx 2023-12-03T11:55:51.900 INFO:tasks.workunit.client.0.smithi116.stderr: -472> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -471> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -470> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -469> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -468> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -467> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -466> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -465> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -464> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -463> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -462> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: crc 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -461> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241ac20) adding con mode: secure 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -460> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -459> 2023-12-03T11:55:51.779+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -458> 2023-12-03T11:55:51.779+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2241ad68 EEXIST 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -457> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.901 INFO:tasks.workunit.client.0.smithi116.stderr: -456> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -455> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7aee350 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -454> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7aee730 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -453> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7ae8a10 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -452> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -451> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -450> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -449> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -448> 2023-12-03T11:55:51.779+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.085546s 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -447> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7aee350 auth_method 0 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -446> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -445> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -444> 2023-12-03T11:55:51.779+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -443> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7aee730 auth_method 0 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -442> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.902 INFO:tasks.workunit.client.0.smithi116.stderr: -441> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -440> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -439> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -438> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -437> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -436> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7ae8a10 auth_method 0 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -435> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -434> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -433> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -432> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -431> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -430> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -429> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -428> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -427> 2023-12-03T11:55:51.780+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -426> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient(hunting): handle_auth_done global_id 7320 payload 274 2023-12-03T11:55:51.903 INFO:tasks.workunit.client.0.smithi116.stderr: -425> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -424> 2023-12-03T11:55:51.780+0000 7fb1582dd640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -423> 2023-12-03T11:55:51.780+0000 7fb1582dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -422> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -421> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -420> 2023-12-03T11:55:51.780+0000 7fb155ad8640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -419> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -418> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr: -417> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: dump: 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -416> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -415> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -414> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -413> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -412> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr: -411> 2023-12-03T11:55:51.780+0000 7fb14e2d9640 10 monclient: dump: 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.905 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -410> 2023-12-03T11:55:51.780+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7320 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -409> 2023-12-03T11:55:51.780+0000 7fb159fdc800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -408> 2023-12-03T11:55:51.780+0000 7fb159fdc800 4 set_mon_vals no callback set 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -407> 2023-12-03T11:55:51.780+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb138069820 auth_method 0 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -406> 2023-12-03T11:55:51.780+0000 7fb159fdc800 10 monclient: shutdown 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -405> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -404> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr: -403> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.906 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -402> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -401> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -400> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding auth protocol: cephx 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -399> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -398> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -397> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -396> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -395> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -394> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.907 INFO:tasks.workunit.client.0.smithi116.stderr: -393> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -392> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -391> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -390> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -389> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: crc 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -388> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab9980) adding con mode: secure 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -387> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -386> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: starting msgr at 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -385> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: starting objecter 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -384> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command objecter_requests cmddesc objecter_requests hook 0x55cba7ae9f00 EEXIST 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -383> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: setting wanted keys 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -382> 2023-12-03T11:55:51.781+0000 7fb159fdc800 1 librados: calling monclient init 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -381> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -380> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -379> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -378> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding auth protocol: cephx 2023-12-03T11:55:51.908 INFO:tasks.workunit.client.0.smithi116.stderr: -377> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -376> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -375> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -374> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -373> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -372> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -371> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -370> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -369> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -368> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -367> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: crc 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -366> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 AuthRegistry(0x55cba7ab81b8) adding con mode: secure 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -365> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -364> 2023-12-03T11:55:51.781+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -363> 2023-12-03T11:55:51.781+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x55cba7ab8300 EEXIST 2023-12-03T11:55:51.909 INFO:tasks.workunit.client.0.smithi116.stderr: -362> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -361> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -360> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7ae8a10 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -359> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7aee350 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -358> 2023-12-03T11:55:51.781+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7ae6830 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -357> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -356> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -355> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -354> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -353> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.087692s 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -352> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7aee350 auth_method 0 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -351> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -350> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -349> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -348> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7ae6830 auth_method 0 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -347> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.910 INFO:tasks.workunit.client.0.smithi116.stderr: -346> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -345> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -344> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -343> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -342> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -341> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7ae8a10 auth_method 0 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -340> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -339> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -338> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -337> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -336> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -335> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -334> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -333> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -332> 2023-12-03T11:55:51.782+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.911 INFO:tasks.workunit.client.0.smithi116.stderr: -331> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient(hunting): handle_auth_done global_id 6896 payload 824 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -330> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -329> 2023-12-03T11:55:51.782+0000 7fb14f2db640 1 monclient: found mon.noname-c 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -328> 2023-12-03T11:55:51.782+0000 7fb14f2db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -327> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -326> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: -325> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: dump: 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.912 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -324> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -323> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -322> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -321> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -320> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: -319> 2023-12-03T11:55:51.782+0000 7fb14cad6640 10 monclient: dump: 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.913 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -318> 2023-12-03T11:55:51.782+0000 7fb159fdc800 5 monclient: authenticate success, global_id 6896 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -317> 2023-12-03T11:55:51.782+0000 7fb14e2d9640 4 set_mon_vals no callback set 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -316> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -315> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -314> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -313> 2023-12-03T11:55:51.782+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -312> 2023-12-03T11:55:51.782+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -311> 2023-12-03T11:55:51.783+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -310> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -309> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -308> 2023-12-03T11:55:51.784+0000 7fb14cad6640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -307> 2023-12-03T11:55:51.784+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb130069820 auth_method 0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -306> 2023-12-03T11:55:51.784+0000 7fb1582dd640 10 monclient: get_auth_request con 0x55cba7b06b80 auth_method 0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -305> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.914 INFO:tasks.workunit.client.0.smithi116.stderr: -304> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 realm 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -303> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -302> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -301> 2023-12-03T11:55:51.785+0000 7fb159fdc800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -300> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -299> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -298> 2023-12-03T11:55:51.785+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -297> 2023-12-03T11:55:51.786+0000 7fb157adc640 10 monclient: get_auth_request con 0x55cba7b0a910 auth_method 0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -296> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -295> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -294> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -293> 2023-12-03T11:55:51.786+0000 7fb159fdc800 20 searching for the correct realm 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -292> 2023-12-03T11:55:51.786+0000 7fb14f2db640 10 monclient: get_auth_request con 0x55cba7b0f850 auth_method 0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -291> 2023-12-03T11:55:51.787+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb140058130 auth_method 0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -290> 2023-12-03T11:55:51.788+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb15007b980 auth_method 0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -289> 2023-12-03T11:55:51.788+0000 7fb14f2db640 10 monclient: get_auth_request con 0x7fb14805e040 auth_method 0 2023-12-03T11:55:51.915 INFO:tasks.workunit.client.0.smithi116.stderr: -288> 2023-12-03T11:55:51.790+0000 7fb1582dd640 10 monclient: get_auth_request con 0x7fb1400573c0 auth_method 0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -287> 2023-12-03T11:55:51.792+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb14005fc90 auth_method 0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -286> 2023-12-03T11:55:51.795+0000 7fb157adc640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55cba7b0a910 msgr2=0x55cba7b0cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -285> 2023-12-03T11:55:51.795+0000 7fb14cad6640 1 client.6896.objecter ms_handle_reset 0x55cba7b0a910 session 0x55cba7877320 osd.5 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -284> 2023-12-03T11:55:51.795+0000 7fb14cad6640 10 monclient: _renew_subs 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -283> 2023-12-03T11:55:51.795+0000 7fb14cad6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -282> 2023-12-03T11:55:51.795+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb130079e10 auth_method 0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -281> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -280> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -279> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -278> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -277> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -276> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -275> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -274> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.916 INFO:tasks.workunit.client.0.smithi116.stderr: -273> 2023-12-03T11:55:51.797+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -272> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -271> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zone default found 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -270> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Realm: () 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -269> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -268> 2023-12-03T11:55:51.798+0000 7fb159fdc800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -267> 2023-12-03T11:55:51.798+0000 7fb159fdc800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -266> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 zonegroup default 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -265> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -264> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -263> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -262> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -261> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -260> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -259> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started sync module instance, tier type = 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -258> 2023-12-03T11:55:51.798+0000 7fb159fdc800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.917 INFO:tasks.workunit.client.0.smithi116.stderr: -257> 2023-12-03T11:55:51.803+0000 7fb1582dd640 0 -- 172.21.15.116:0/1335896381 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fb1400573c0 msgr2=0x7fb1400769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -256> 2023-12-03T11:55:51.803+0000 7fb14cad6640 1 client.6896.objecter ms_handle_reset 0x7fb1400573c0 session 0x7fb140060120 osd.6 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -255> 2023-12-03T11:55:51.804+0000 7fb1582dd640 10 monclient: get_auth_request con 0x55cba7b0a910 auth_method 0 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -254> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=0 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -253> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=1 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -252> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=2 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -251> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=3 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -250> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=4 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -249> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=5 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -248> 2023-12-03T11:55:51.807+0000 7fb159fdc800 20 add_watcher() i=6 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -247> 2023-12-03T11:55:51.808+0000 7fb159fdc800 20 add_watcher() i=7 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -246> 2023-12-03T11:55:51.808+0000 7fb159fdc800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -245> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -244> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -243> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding auth protocol: cephx 2023-12-03T11:55:51.918 INFO:tasks.workunit.client.0.smithi116.stderr: -242> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -241> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -240> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -239> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -238> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -237> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -236> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -235> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -234> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -233> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -232> 2023-12-03T11:55:51.809+0000 7fb11dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -231> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: crc 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -230> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b650) adding con mode: secure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -229> 2023-12-03T11:55:51.809+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -228> 2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -227> 2023-12-03T11:55:51.809+0000 7fb159fdc800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.919 INFO:tasks.workunit.client.0.smithi116.stderr: -226> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache list hook 0x55cba7a94700 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -225> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache inspect hook 0x55cba7a94700 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -224> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache erase hook 0x55cba7a94700 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -223> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cache zap hook 0x55cba7a94700 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -222> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: get_monmap_and_config 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -221> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -220> 2023-12-03T11:55:51.809+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr: -219> 2023-12-03T11:55:51.809+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.920 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -218> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -217> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -216> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding auth protocol: cephx 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -215> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -214> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -213> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -212> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -211> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -210> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -209> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -208> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -207> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -206> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.921 INFO:tasks.workunit.client.0.smithi116.stderr: -205> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: crc 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -204> 2023-12-03T11:55:51.809+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4b030) adding con mode: secure 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -203> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -202> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -201> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -200> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -199> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding auth protocol: cephx 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -198> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -197> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -196> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -195> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -194> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -193> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -192> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -191> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.922 INFO:tasks.workunit.client.0.smithi116.stderr: -190> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -189> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -188> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: crc 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -187> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 AuthRegistry(0x7ffc2241b1a0) adding con mode: secure 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -186> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -185> 2023-12-03T11:55:51.810+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -184> 2023-12-03T11:55:51.810+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2241b2e8 EEXIST 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -183> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -182> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -181> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7b49770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -180> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7b474c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -179> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7b40610 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -178> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -177> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -176> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -175> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.923 INFO:tasks.workunit.client.0.smithi116.stderr: -174> 2023-12-03T11:55:51.810+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.116432s 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -173> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7b49770 auth_method 0 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -172> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -171> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -170> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -169> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7b474c0 auth_method 0 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -168> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -167> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -166> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -165> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -164> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -163> 2023-12-03T11:55:51.810+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -162> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -161> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -160> 2023-12-03T11:55:51.810+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -159> 2023-12-03T11:55:51.810+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7b40610 auth_method 0 2023-12-03T11:55:51.924 INFO:tasks.workunit.client.0.smithi116.stderr: -158> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -157> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -156> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -155> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -154> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -153> 2023-12-03T11:55:51.811+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -152> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient(hunting): handle_auth_done global_id 7326 payload 274 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -151> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -150> 2023-12-03T11:55:51.811+0000 7fb157adc640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -149> 2023-12-03T11:55:51.811+0000 7fb157adc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -148> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -147> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr: -146> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: dump: 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.925 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -145> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -144> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -143> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -142> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -141> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.926 INFO:tasks.workunit.client.0.smithi116.stderr: -140> 2023-12-03T11:55:51.811+0000 7fb11d7e2640 10 monclient: dump: 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -139> 2023-12-03T11:55:51.811+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7326 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -138> 2023-12-03T11:55:51.811+0000 7fb159fdc800 10 monclient: get_monmap_and_config success 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -137> 2023-12-03T11:55:51.811+0000 7fb159fdc800 4 set_mon_vals no callback set 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -136> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: shutdown 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -135> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: build_initial_monmap 2023-12-03T11:55:51.927 INFO:tasks.workunit.client.0.smithi116.stderr: -134> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 build_initial for_mkfs: 0 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -133> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: monmap: 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 0 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 0.000000 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:created 0.000000 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 0 (unknown) 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.noname-b 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -132> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -131> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -130> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding auth protocol: cephx 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -129> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.928 INFO:tasks.workunit.client.0.smithi116.stderr: -128> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -127> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -126> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -125> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -124> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -123> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -122> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -121> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -120> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -119> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: crc 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -118> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b40b30) adding con mode: secure 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -117> 2023-12-03T11:55:51.812+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -116> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: starting msgr at 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -115> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: starting objecter 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -114> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command objecter_requests cmddesc objecter_requests hook 0x55cba7b14560 EEXIST 2023-12-03T11:55:51.929 INFO:tasks.workunit.client.0.smithi116.stderr: -113> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: setting wanted keys 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -112> 2023-12-03T11:55:51.812+0000 7fb159fdc800 1 librados: calling monclient init 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -111> 2023-12-03T11:55:51.812+0000 7fb159fdc800 10 monclient: init 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -110> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -109> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -108> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding auth protocol: cephx 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -107> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -106> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -105> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -104> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -103> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -102> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -101> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -100> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -99> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -98> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.930 INFO:tasks.workunit.client.0.smithi116.stderr: -97> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: crc 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -96> 2023-12-03T11:55:51.812+0000 7fb159fdc800 5 AuthRegistry(0x55cba7b4e548) adding con mode: secure 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -95> 2023-12-03T11:55:51.812+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -94> 2023-12-03T11:55:51.813+0000 7fb159fdc800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -93> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command rotate-key cmddesc rotate-key hook 0x55cba7b4e690 EEXIST 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -92> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _reopen_session rank -1 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -91> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -90> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-a con 0x55cba7b40610 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -89> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-b con 0x55cba7b474c0 addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -88> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): picked mon.noname-c con 0x55cba7b49770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -87> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -86> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -85> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): start opening mon connection 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -84> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): _renew_subs 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -83> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient(hunting): authenticate will time out at 1608.118842s 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -82> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): get_auth_request con 0x55cba7b40610 auth_method 0 2023-12-03T11:55:51.931 INFO:tasks.workunit.client.0.smithi116.stderr: -81> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -80> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -79> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -78> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): get_auth_request con 0x55cba7b474c0 auth_method 0 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -77> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -76> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -75> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -74> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): get_auth_request con 0x55cba7b49770 auth_method 0 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -73> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -72> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -71> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -70> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -69> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -68> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -67> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -66> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.932 INFO:tasks.workunit.client.0.smithi116.stderr: -65> 2023-12-03T11:55:51.813+0000 7fb14f2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -64> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -63> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -62> 2023-12-03T11:55:51.813+0000 7fb157adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -61> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient(hunting): handle_auth_done global_id 7329 payload 824 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -60> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient: _finish_hunting 0 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -59> 2023-12-03T11:55:51.813+0000 7fb1582dd640 1 monclient: found mon.noname-a 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -58> 2023-12-03T11:55:51.813+0000 7fb1582dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -57> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -56> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr: -55> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: dump: 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -54> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: _finish_auth 0 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -53> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: _check_auth_tickets 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -52> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -51> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -50> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr: -49> 2023-12-03T11:55:51.813+0000 7fb0977fe640 10 monclient: dump: 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:epoch 1 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:fsid 1fb29ff4-87be-4f29-84b9-4dc87a2589ed 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:last_changed 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:created 2023-12-03T11:41:33.709349+0000 2023-12-03T11:55:51.934 INFO:tasks.workunit.client.0.smithi116.stderr:min_mon_release 18 (reef) 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr:election_strategy: 1 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr:1: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -48> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 monclient: authenticate success, global_id 7329 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -47> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -46> 2023-12-03T11:55:51.813+0000 7fb11d7e2640 4 set_mon_vals no callback set 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -45> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -44> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _renew_subs 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -43> 2023-12-03T11:55:51.813+0000 7fb159fdc800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -42> 2023-12-03T11:55:51.813+0000 7fb159fdc800 1 librados: init done 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -41> 2023-12-03T11:55:51.813+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command cr dump hook 0x55cba7b4fd28 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -40> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc handle_mgr_map Got map version 5 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -39> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.935 INFO:tasks.workunit.client.0.smithi116.stderr: -38> 2023-12-03T11:55:51.815+0000 7fb0977fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.152:6832/2965493816,v1:172.21.15.152:6833/2965493816] 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -37> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 note: GC not initialized 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -36> 2023-12-03T11:55:51.815+0000 7fb157adc640 10 monclient: get_auth_request con 0x7fb08c069820 auth_method 0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -35> 2023-12-03T11:55:51.815+0000 7fb094ff9640 20 reqs_thread_entry: start 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -34> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace show hook 0x7fb1500047e0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -33> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace history hook 0x7fb1500047e0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -32> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace active hook 0x7fb1500047e0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -31> 2023-12-03T11:55:51.815+0000 7fb159fdc800 5 asok(0x55cba78aea10) register_command sync trace active_short hook 0x7fb1500047e0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -30> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -29> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 Filter name: none 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -28> 2023-12-03T11:55:51.815+0000 7fb08a7fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -27> 2023-12-03T11:55:51.815+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -26> 2023-12-03T11:55:51.815+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -25> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -24> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -23> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:55:51.936 INFO:tasks.workunit.client.0.smithi116.stderr: -22> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 : miss 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -21> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -20> 2023-12-03T11:55:51.816+0000 7fb159fdc800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -19> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -18> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -17> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -16> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -15> 2023-12-03T11:55:51.816+0000 7fb159fdc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -14> 2023-12-03T11:55:51.816+0000 7fb14f2db640 10 monclient: get_auth_request con 0x55cba7b9b1c0 auth_method 0 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -13> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -12> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -11> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -10> 2023-12-03T11:55:51.821+0000 7fb159fdc800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27, ofs0, ns] 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -9> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660546 cookie 140399528550400 notifier 6896 bl.length()=873 2023-12-03T11:55:51.937 INFO:tasks.workunit.client.0.smithi116.stderr: -8> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -7> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -6> 2023-12-03T11:55:51.822+0000 7fb14dad8640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -5> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_sources= new_sources= 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -4> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential sources added=[] removed=[] 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -3> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_dests= new_dests= 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -2> 2023-12-03T11:55:51.822+0000 7fb159fdc800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential dests added=[] removed=[] 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: -1> 2023-12-03T11:55:51.825+0000 7fb159fdc800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fb159fdc800 time 2023-12-03T11:55:51.825730+0000 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.938 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fb15e96eef8] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 5: main() 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 0> 2023-12-03T11:55:51.826+0000 7fb159fdc800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: in thread 7fb159fdc800 thread_name:radosgw-admin 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb15c254db0] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fb15c2a154c] 2023-12-03T11:55:51.939 INFO:tasks.workunit.client.0.smithi116.stderr: 3: raise() 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 4: abort() 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fb15e96ef52] 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fb15e96f0b6] 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cba6936a71] 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x55cba6942f34] 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 9: main() 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb15c23feb0] 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 11: __libc_start_main() 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 12: _start() 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr:--- logging levels --- 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 none 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 lockdep 2023-12-03T11:55:51.940 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 context 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 crush 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_balancer 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_locker 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_log_expire 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mds_migrator 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 buffer 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 timer 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 filer 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 striper 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 1 objecter 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rados 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd 2023-12-03T11:55:51.941 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_mirror 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_replay 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 rbd_pwl 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 journaler 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objectcacher 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 client 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 osd 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 optracker 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 objclass 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 filestore 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 journal 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 ms 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mon 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 0/10 monc 2023-12-03T11:55:51.942 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 paxos 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 tp 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 auth 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 crypto 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 finisher 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 reserver 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 heartbeatmap 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 perfcounter 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 20/20 rgw 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_sync 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_datacache 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_access 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 rgw_flight 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 javaclient 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 asok 2023-12-03T11:55:51.943 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 1 throttle 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 0 refs 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 compressor 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluestore 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 bluefs 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 3 bdev 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 kstore 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 4/ 5 rocksdb 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 fuse 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 2/ 5 mgr 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mgrc 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 dpdk 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 eventtrace 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 prioritycache 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 test 2023-12-03T11:55:51.944 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cephsqlite 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_onode 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_odata 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_omap 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_tm 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_t 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_epm 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_lba 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_cache 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_journal 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_device 2023-12-03T11:55:51.945 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 seastore_backref 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 alienstore 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 mclock 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 0/ 5 cyanstore 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 ceph_exporter 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 memstore 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 1/ 5 trace 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: -2/-2 (syslog threshold) 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 99/99 (stderr threshold) 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb08a7fc640 / http_manager 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb094ff9640 / http_manager 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb0977fe640 / ms_dispatch 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb11d7e2640 / io_context_pool 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb11dfe3640 / rgw_dt_lg_renew 2023-12-03T11:55:51.946 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14cad6640 / ms_dispatch 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14dad8640 / io_context_pool 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14e2d9640 / io_context_pool 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb14f2db640 / msgr-worker-2 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb155ad8640 / ms_dispatch 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb157adc640 / msgr-worker-1 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb1582dd640 / msgr-worker-0 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: 7fb159fdc800 / radosgw-admin 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: max_recent 500 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: max_new 1000 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:55:51.827071Z_bf67508e-25ba-4e6c-ab30-1de3ca74f0de/log 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr:--- end dump of recent events --- 2023-12-03T11:55:51.947 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:55:51.952 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:55:51.952 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:55:51.952 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.872+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.952 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 20 realm 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.873+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.874+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.874+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.874+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.874+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.875+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.875+0000 7f42ddf31800 20 searching for the correct realm 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.883+0000 7f42d9bfb640 0 -- 172.21.15.116:0/2262502761 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f42d002e370 msgr2=0x7f42d002e7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:55:51.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.886+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 zone default found 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 4 Realm: () 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 zonegroup default 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.887+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.888+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.888+0000 7f42ddf31800 20 started sync module instance, tier type = 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.888+0000 7f42ddf31800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.888+0000 7f42da3fc640 0 -- 172.21.15.116:0/2262502761 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f42d001b480 msgr2=0x7f42d002f3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=0 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=1 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=2 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=3 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=4 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=5 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=6 2023-12-03T11:55:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 20 add_watcher() i=7 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.897+0000 7f42ddf31800 2 all 8 watchers are set, enabling cache 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.899+0000 7f429f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.899+0000 7f42ddf31800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.899+0000 7f42ddf31800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.905+0000 7f42ddf31800 5 note: GC not initialized 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.905+0000 7f420e7fc640 20 reqs_thread_entry: start 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f42ddf31800 20 init_complete bucket index max shards: 11 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f42ddf31800 20 Filter name: none 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f41fffff640 20 reqs_thread_entry: start 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.906+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=403 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 rados_obj.operate() r=0 bl.length=403 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.907+0000 7f42ddf31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.908+0000 7f42ddf31800 20 rados->read ofs=0 len=0 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.908+0000 7f42ddf31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.909+0000 7f429f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.911+0000 7f42ddf31800 20 remove_watcher() i=0 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.911+0000 7f42ddf31800 2 removed watcher, disabling cache 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.912+0000 7f42ddf31800 20 remove_watcher() i=1 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.913+0000 7f42ddf31800 20 remove_watcher() i=2 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.914+0000 7f42ddf31800 20 remove_watcher() i=3 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.915+0000 7f42ddf31800 20 remove_watcher() i=4 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.916+0000 7f42ddf31800 20 remove_watcher() i=5 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.917+0000 7f42ddf31800 20 remove_watcher() i=6 2023-12-03T11:55:51.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:55:51.918+0000 7f42ddf31800 20 remove_watcher() i=7 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 num_objects: 20 size_kb: 1 num_shards: 11 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:55:51.958 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:21.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:21.963 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.007 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 12 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:56:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.055 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.054+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 realm 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.056+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.057+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.057+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.057+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.059 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.057+0000 7f803b00f800 20 searching for the correct realm 2023-12-03T11:56:22.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.069+0000 7f80365b1640 0 -- 172.21.15.116:0/3715052420 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x563c4f322bc0 msgr2=0x563c4f324fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.070+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 zone default found 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 4 Realm: () 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 zonegroup default 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.071+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.072+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.072+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.072+0000 7f803b00f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.072+0000 7f803b00f800 20 started sync module instance, tier type = 2023-12-03T11:56:22.075 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.072+0000 7f803b00f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.079+0000 7f8036db2640 0 -- 172.21.15.116:0/3715052420 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f802005e920 msgr2=0x7f802007d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=0 2023-12-03T11:56:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=1 2023-12-03T11:56:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=2 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=3 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=4 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=5 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.082+0000 7f803b00f800 20 add_watcher() i=6 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.083+0000 7f803b00f800 20 add_watcher() i=7 2023-12-03T11:56:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.083+0000 7f803b00f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.085+0000 7f7ffbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.085+0000 7f803b00f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.085+0000 7f803b00f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.091+0000 7f803b00f800 5 note: GC not initialized 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.091+0000 7f7f6affd640 20 reqs_thread_entry: start 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.091+0000 7f803b00f800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.091+0000 7f803b00f800 20 Filter name: none 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.091+0000 7f7f68ff9640 20 reqs_thread_entry: start 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:56:22.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:56:22.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:56:22.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:56:22.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 : miss 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.092+0000 7f803b00f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:56:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.093+0000 7f803b00f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 2023-12-03T11:56:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.098+0000 7f803b00f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:56:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.098+0000 7f803b00f800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:56:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.098+0000 7f803b00f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:56:22.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.098+0000 7f803b00f800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27, ofs0, ns] 2023-12-03T11:56:22.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.282+0000 7f802cff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660548 cookie 140188538077184 notifier 6932 bl.length()=873 2023-12-03T11:56:22.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.282+0000 7f802cff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.282+0000 7f802cff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.282+0000 7f802cff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.283+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_sources= new_sources= 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.283+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential sources added=[] removed=[] 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.283+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_dests= new_dests= 2023-12-03T11:56:22.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.283+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential dests added=[] removed=[] 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.0 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.0 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.1 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.1 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.2 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.2 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.3 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.3 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.4 2023-12-03T11:56:22.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.4 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.5 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.5 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.6 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.6 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.7 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.7 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.8 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.8 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.9 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.9 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.10 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.10 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.11 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1.11 2023-12-03T11:56:22.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.285+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.0 2023-12-03T11:56:22.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.286+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.1 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.286+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.2 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.287+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.3 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.287+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.4 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.287+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.5 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.288+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.6 2023-12-03T11:56:22.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.288+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.7 2023-12-03T11:56:22.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.289+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.8 2023-12-03T11:56:22.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.290+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.9 2023-12-03T11:56:22.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.291+0000 7f803b00f800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.27.10 2023-12-03T11:56:22.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.294+0000 7f803b00f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 : hit (requested=0x17, cached=0x17) 2023-12-03T11:56:22.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.294+0000 7f803b00f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.295+0000 7f803b00f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.295+0000 7f803b00f800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.295+0000 7f803b00f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.295+0000 7f803b00f800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27, ofs0, ns] 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f802cff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660549 cookie 140188538077184 notifier 6932 bl.length()=841 2023-12-03T11:56:22.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f802cff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 info.flags=0x17 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f802cff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 to cache LRU end 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f802cff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_sources= new_sources= 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential sources added=[] removed=[] 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): orig_dests= new_dests= 2023-12-03T11:56:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.296+0000 7f803b00f800 20 handle_bi_update(): bucket=:abort-at-do-reshard[21f2df23-11b5-4c91-821f-a8e55156195e.4668.27]): potential dests added=[] removed=[] 2023-12-03T11:56:22.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.298+0000 7f803b00f800 1 execute INFO: reshard of bucket "abort-at-do-reshard" completed successfully 2023-12-03T11:56:22.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.300+0000 7f7ffbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.302+0000 7f803b00f800 20 remove_watcher() i=0 2023-12-03T11:56:22.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.302+0000 7f803b00f800 2 removed watcher, disabling cache 2023-12-03T11:56:22.305 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.303+0000 7f803b00f800 20 remove_watcher() i=1 2023-12-03T11:56:22.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.305+0000 7f803b00f800 20 remove_watcher() i=2 2023-12-03T11:56:22.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.306+0000 7f803b00f800 20 remove_watcher() i=3 2023-12-03T11:56:22.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.307+0000 7f803b00f800 20 remove_watcher() i=4 2023-12-03T11:56:22.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.308+0000 7f803b00f800 20 remove_watcher() i=5 2023-12-03T11:56:22.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.309+0000 7f803b00f800 20 remove_watcher() i=6 2023-12-03T11:56:22.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.310+0000 7f803b00f800 20 remove_watcher() i=7 2023-12-03T11:56:22.317 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:56:22.344 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.345 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.357+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 realm 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.361 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.359+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.360+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.360+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.360+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.360+0000 7f5adec0f800 20 searching for the correct realm 2023-12-03T11:56:22.372 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.370+0000 7f5ada28c640 0 -- 172.21.15.116:0/1097589570 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x561600416820 msgr2=0x561600418c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.373 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.374 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.372+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 zone default found 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 4 Realm: () 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 zonegroup default 2023-12-03T11:56:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 started sync module instance, tier type = 2023-12-03T11:56:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.373+0000 7f5adec0f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.380+0000 7f5adaa8d640 0 -- 172.21.15.116:0/1097589570 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f5ad005ef10 msgr2=0x7f5ad0081790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=0 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=1 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=2 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=3 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=4 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=5 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.382+0000 7f5adec0f800 20 add_watcher() i=6 2023-12-03T11:56:22.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.383+0000 7f5adec0f800 20 add_watcher() i=7 2023-12-03T11:56:22.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.383+0000 7f5adec0f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.385+0000 7f5a9ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.385+0000 7f5adec0f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.385+0000 7f5adec0f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5adec0f800 5 note: GC not initialized 2023-12-03T11:56:22.392 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5a0effd640 20 reqs_thread_entry: start 2023-12-03T11:56:22.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5adec0f800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5adec0f800 20 Filter name: none 2023-12-03T11:56:22.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5a0cff9640 20 reqs_thread_entry: start 2023-12-03T11:56:22.393 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.391+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.392+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.393+0000 7f5adec0f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:56:22.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.393+0000 7f5adec0f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.394+0000 7f5adec0f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.395 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.394+0000 7f5adec0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.395+0000 7f5a9ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.397+0000 7f5adec0f800 20 remove_watcher() i=0 2023-12-03T11:56:22.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.397+0000 7f5adec0f800 2 removed watcher, disabling cache 2023-12-03T11:56:22.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.398+0000 7f5adec0f800 20 remove_watcher() i=1 2023-12-03T11:56:22.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.399+0000 7f5adec0f800 20 remove_watcher() i=2 2023-12-03T11:56:22.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.400+0000 7f5adec0f800 20 remove_watcher() i=3 2023-12-03T11:56:22.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.401+0000 7f5adec0f800 20 remove_watcher() i=4 2023-12-03T11:56:22.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.402+0000 7f5adec0f800 20 remove_watcher() i=5 2023-12-03T11:56:22.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.403+0000 7f5adec0f800 20 remove_watcher() i=6 2023-12-03T11:56:22.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.404+0000 7f5adec0f800 20 remove_watcher() i=7 2023-12-03T11:56:22.410 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.410 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:56:22.410 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: abort-at-do-reshard id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.27 num_objects: 19 size_kb: 1 num_shards: 12 2023-12-03T11:56:22.410 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.412 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.415 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.459 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.485 INFO:tasks.workunit.client.0.smithi116.stderr: test: reshard versioned bucket 2023-12-03T11:56:22.487 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:56:22.513 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.513 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.525+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.529 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 realm 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.527+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.528+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.528+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.529+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.529+0000 7f2ed5e12800 20 searching for the correct realm 2023-12-03T11:56:22.539 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.537+0000 7f2ed1457640 0 -- 172.21.15.116:0/3403590960 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2ecc02f4b0 msgr2=0x7f2ecc02f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:56:22.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.541 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.539+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 zone default found 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 4 Realm: () 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 zonegroup default 2023-12-03T11:56:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.540+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.541+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.541+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.541+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.541+0000 7f2ed5e12800 20 started sync module instance, tier type = 2023-12-03T11:56:22.546 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.541+0000 7f2ed5e12800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.547 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.545+0000 7f2ed1457640 0 -- 172.21.15.116:0/3403590960 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2eac079d90 msgr2=0x7f2eac07c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=0 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=1 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=2 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=3 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=4 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=5 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.549+0000 7f2ed5e12800 20 add_watcher() i=6 2023-12-03T11:56:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.550+0000 7f2ed5e12800 20 add_watcher() i=7 2023-12-03T11:56:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.550+0000 7f2ed5e12800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.552+0000 7f2e96fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.552+0000 7f2ed5e12800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.552+0000 7f2ed5e12800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2ed5e12800 5 note: GC not initialized 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2e05ffb640 20 reqs_thread_entry: start 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2ed5e12800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2ed5e12800 20 Filter name: none 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2df77fe640 20 reqs_thread_entry: start 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.558+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.559+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.559+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.559+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.559+0000 7f2ed5e12800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.559+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.560+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.560+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.560+0000 7f2ed5e12800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:22.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.560+0000 7f2ed5e12800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.561+0000 7f2ed5e12800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.561+0000 7f2ed5e12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.563 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.562+0000 7f2e96fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.564+0000 7f2ed5e12800 20 remove_watcher() i=0 2023-12-03T11:56:22.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.564+0000 7f2ed5e12800 2 removed watcher, disabling cache 2023-12-03T11:56:22.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.565+0000 7f2ed5e12800 20 remove_watcher() i=1 2023-12-03T11:56:22.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.566+0000 7f2ed5e12800 20 remove_watcher() i=2 2023-12-03T11:56:22.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.567+0000 7f2ed5e12800 20 remove_watcher() i=3 2023-12-03T11:56:22.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.568+0000 7f2ed5e12800 20 remove_watcher() i=4 2023-12-03T11:56:22.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.569+0000 7f2ed5e12800 20 remove_watcher() i=5 2023-12-03T11:56:22.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.570+0000 7f2ed5e12800 20 remove_watcher() i=6 2023-12-03T11:56:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.571+0000 7f2ed5e12800 20 remove_watcher() i=7 2023-12-03T11:56:22.576 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.576 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:56:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: myver id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 num_objects: 0 size_kb: 0 num_shards: 11 2023-12-03T11:56:22.577 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 12 2023-12-03T11:56:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.620+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 realm 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.622+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.623+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.623+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.623+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.623+0000 7f1146524800 20 searching for the correct realm 2023-12-03T11:56:22.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.631+0000 7f1142a5b640 0 -- 172.21.15.116:0/3451931075 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f112c021040 msgr2=0x7f112c017950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.635 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.636 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.634+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 zone default found 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 4 Realm: () 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 zonegroup default 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1139a59640 0 -- 172.21.15.116:0/3451931075 >> [v2:172.21.15.152:6817/2654994651,v1:172.21.15.152:6819/2654994651] conn(0x7f11340a0c40 msgr2=0x7f1134097de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.635+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.636+0000 7f1146524800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.636+0000 7f1146524800 20 started sync module instance, tier type = 2023-12-03T11:56:22.638 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.636+0000 7f1146524800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=0 2023-12-03T11:56:22.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=1 2023-12-03T11:56:22.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=2 2023-12-03T11:56:22.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=3 2023-12-03T11:56:22.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=4 2023-12-03T11:56:22.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=5 2023-12-03T11:56:22.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.644+0000 7f1146524800 20 add_watcher() i=6 2023-12-03T11:56:22.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.645+0000 7f1146524800 20 add_watcher() i=7 2023-12-03T11:56:22.646 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.645+0000 7f1146524800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.647 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.646+0000 7f1107fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.646+0000 7f1146524800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.646+0000 7f1146524800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.652+0000 7f1146524800 5 note: GC not initialized 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.652+0000 7f1076ffd640 20 reqs_thread_entry: start 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.652+0000 7f1146524800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 20 Filter name: none 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1074ff9640 20 reqs_thread_entry: start 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : miss 2023-12-03T11:56:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.653+0000 7f1146524800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.654+0000 7f1146524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.654+0000 7f1146524800 10 adding default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.654+0000 7f1146524800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.654+0000 7f1146524800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:22.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.654+0000 7f1146524800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1138a57640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660540 cookie 94603953747264 notifier 6989 bl.length()=783 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1138a57640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1138a57640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1138a57640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:22.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.660+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:22.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.0 2023-12-03T11:56:22.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.0 2023-12-03T11:56:22.665 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.1 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.1 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.2 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.2 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.3 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.3 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.4 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.4 2023-12-03T11:56:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.5 2023-12-03T11:56:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.5 2023-12-03T11:56:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.6 2023-12-03T11:56:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.6 2023-12-03T11:56:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.7 2023-12-03T11:56:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.7 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.8 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.8 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.9 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.9 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.10 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.10 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.11 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.11 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.0 2023-12-03T11:56:22.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.663+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1 2023-12-03T11:56:22.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.664+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2 2023-12-03T11:56:22.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.664+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.665+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.4 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.665+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.5 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.665+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.6 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.666+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.7 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.666+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.8 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.666+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.9 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.667+0000 7f1146524800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.10 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.667+0000 7f1146524800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:56:22.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.667+0000 7f1146524800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.668+0000 7f1146524800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.668+0000 7f1146524800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.668+0000 7f1146524800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.668+0000 7f1146524800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1123fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660541 cookie 94603953747264 notifier 6989 bl.length()=751 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1123fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1123fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1123fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:22.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:22.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.669+0000 7f1146524800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.671+0000 7f1146524800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:56:22.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.673+0000 7f1107fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.675+0000 7f1146524800 20 remove_watcher() i=0 2023-12-03T11:56:22.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.675+0000 7f1146524800 2 removed watcher, disabling cache 2023-12-03T11:56:22.678 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.677+0000 7f1146524800 20 remove_watcher() i=1 2023-12-03T11:56:22.679 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.678+0000 7f1146524800 20 remove_watcher() i=2 2023-12-03T11:56:22.680 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.679+0000 7f1146524800 20 remove_watcher() i=3 2023-12-03T11:56:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.679+0000 7f1146524800 20 remove_watcher() i=4 2023-12-03T11:56:22.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.680+0000 7f1146524800 20 remove_watcher() i=5 2023-12-03T11:56:22.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.681+0000 7f1146524800 20 remove_watcher() i=6 2023-12-03T11:56:22.683 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.682+0000 7f1146524800 20 remove_watcher() i=7 2023-12-03T11:56:22.690 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:56:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.728+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 realm 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.730+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.731+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.731+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.731+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.731+0000 7f3e7c337800 20 searching for the correct realm 2023-12-03T11:56:22.741 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.740+0000 7f3e7806f640 0 -- 172.21.15.116:0/2782718169 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f3e680523e0 msgr2=0x7f3e68052840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.741+0000 7f3e7c337800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.741+0000 7f3e7c337800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.741+0000 7f3e7c337800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.741+0000 7f3e7c337800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.741+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 zone default found 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 4 Realm: () 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 zonegroup default 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.742+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.743+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.743+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.743+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.743+0000 7f3e7c337800 20 started sync module instance, tier type = 2023-12-03T11:56:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.743+0000 7f3e7c337800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.748+0000 7f3e78870640 0 -- 172.21.15.116:0/2782718169 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f3e6c072cb0 msgr2=0x7f3e6c03a8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=0 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=1 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=2 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=3 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=4 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=5 2023-12-03T11:56:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.751+0000 7f3e7c337800 20 add_watcher() i=6 2023-12-03T11:56:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.752+0000 7f3e7c337800 20 add_watcher() i=7 2023-12-03T11:56:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.752+0000 7f3e7c337800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.754+0000 7f3e3e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.754+0000 7f3e7c337800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.754+0000 7f3e7c337800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3e7c337800 5 note: GC not initialized 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3dad7fa640 20 reqs_thread_entry: start 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3e7c337800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3e7c337800 20 Filter name: none 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3da6ffd640 20 reqs_thread_entry: start 2023-12-03T11:56:22.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.760+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.761+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:22.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.761+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.761+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:22.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.761+0000 7f3e7c337800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.761+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.762+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.762+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.762+0000 7f3e7c337800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.762+0000 7f3e7c337800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.763+0000 7f3e7c337800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.763+0000 7f3e7c337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.764+0000 7f3e3e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.766+0000 7f3e7c337800 20 remove_watcher() i=0 2023-12-03T11:56:22.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.766+0000 7f3e7c337800 2 removed watcher, disabling cache 2023-12-03T11:56:22.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.767+0000 7f3e7c337800 20 remove_watcher() i=1 2023-12-03T11:56:22.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.768+0000 7f3e7c337800 20 remove_watcher() i=2 2023-12-03T11:56:22.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.769+0000 7f3e7c337800 20 remove_watcher() i=3 2023-12-03T11:56:22.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.770+0000 7f3e7c337800 20 remove_watcher() i=4 2023-12-03T11:56:22.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.770+0000 7f3e7c337800 20 remove_watcher() i=5 2023-12-03T11:56:22.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.771+0000 7f3e7c337800 20 remove_watcher() i=6 2023-12-03T11:56:22.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.772+0000 7f3e7c337800 20 remove_watcher() i=7 2023-12-03T11:56:22.778 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.778 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:56:22.778 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: myver id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 num_objects: 0 size_kb: 0 num_shards: 12 2023-12-03T11:56:22.779 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:22.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.781 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.783 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.824 INFO:tasks.workunit.client.0.smithi116.stderr: test: reshard removes olh entries with empty name 2023-12-03T11:56:22.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.826 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:22.873 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:22.916 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:56:22.943 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:22.943 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:22.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.956+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 realm 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.958+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.959+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.959+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.959+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:22.961 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.960+0000 7f6258b3b800 20 searching for the correct realm 2023-12-03T11:56:22.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.969+0000 7f6254840640 0 -- 172.21.15.116:0/1116348617 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f624c020020 msgr2=0x7f624c0592f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:22.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:22.971 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:22.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:22.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.970+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 zone default found 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 4 Realm: () 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:22.973 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 zonegroup default 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 started sync module instance, tier type = 2023-12-03T11:56:22.974 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.971+0000 7f6258b3b800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:22.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.976+0000 7f6254840640 0 -- 172.21.15.116:0/1116348617 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f6230079e10 msgr2=0x7f623007c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:22.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=0 2023-12-03T11:56:22.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=1 2023-12-03T11:56:22.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=2 2023-12-03T11:56:22.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=3 2023-12-03T11:56:22.981 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=4 2023-12-03T11:56:22.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=5 2023-12-03T11:56:22.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.980+0000 7f6258b3b800 20 add_watcher() i=6 2023-12-03T11:56:22.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.981+0000 7f6258b3b800 20 add_watcher() i=7 2023-12-03T11:56:22.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.981+0000 7f6258b3b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:22.983 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.982+0000 7f621afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.983+0000 7f6258b3b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:22.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.983+0000 7f6258b3b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:22.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.988+0000 7f6258b3b800 5 note: GC not initialized 2023-12-03T11:56:22.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.988+0000 7f6189ffb640 20 reqs_thread_entry: start 2023-12-03T11:56:22.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.988+0000 7f6258b3b800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:22.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 Filter name: none 2023-12-03T11:56:22.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f617b7fe640 20 reqs_thread_entry: start 2023-12-03T11:56:22.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:22.991 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.989+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.990+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.990+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.990+0000 7f6258b3b800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:22.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.990+0000 7f6258b3b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:22.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.991+0000 7f6258b3b800 20 rados->read ofs=0 len=0 2023-12-03T11:56:22.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.991+0000 7f6258b3b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:22.993 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.992+0000 7f621afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:22.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.994+0000 7f6258b3b800 20 remove_watcher() i=0 2023-12-03T11:56:22.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.994+0000 7f6258b3b800 2 removed watcher, disabling cache 2023-12-03T11:56:22.996 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.995+0000 7f6258b3b800 20 remove_watcher() i=1 2023-12-03T11:56:22.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.996+0000 7f6258b3b800 20 remove_watcher() i=2 2023-12-03T11:56:22.998 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.997+0000 7f6258b3b800 20 remove_watcher() i=3 2023-12-03T11:56:22.999 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.998+0000 7f6258b3b800 20 remove_watcher() i=4 2023-12-03T11:56:23.000 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.999+0000 7f6258b3b800 20 remove_watcher() i=5 2023-12-03T11:56:23.001 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:22.999+0000 7f6258b3b800 20 remove_watcher() i=6 2023-12-03T11:56:23.002 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.000+0000 7f6258b3b800 20 remove_watcher() i=7 2023-12-03T11:56:23.006 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:23.006 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:56:23.006 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: a-bucket id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 num_objects: 0 size_kb: 0 num_shards: 13 2023-12-03T11:56:23.006 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:23.008 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket layout --bucket a-bucket 2023-12-03T11:56:23.035 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.035 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.047+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 20 realm 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.048+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.050 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.049+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.049+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.049+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.049+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.050+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.050+0000 7f2bbce11800 20 searching for the correct realm 2023-12-03T11:56:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.059+0000 7f2bb7bb0640 0 -- 172.21.15.116:0/3109892088 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x562b6687c250 msgr2=0x562b6687e640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.060+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 zone default found 2023-12-03T11:56:23.063 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 4 Realm: () 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 zonegroup default 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 started sync module instance, tier type = 2023-12-03T11:56:23.064 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.061+0000 7f2bbce11800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.068+0000 7f2bb8bb2640 0 -- 172.21.15.116:0/3109892088 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f2bac04cc00 msgr2=0x7f2bac0c80d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=0 2023-12-03T11:56:23.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=1 2023-12-03T11:56:23.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=2 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=3 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=4 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=5 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.070+0000 7f2bbce11800 20 add_watcher() i=6 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.071+0000 7f2bbce11800 20 add_watcher() i=7 2023-12-03T11:56:23.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.071+0000 7f2bbce11800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.072+0000 7f2b7e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.072+0000 7f2bbce11800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.072+0000 7f2bbce11800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2bbce11800 5 note: GC not initialized 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2aed7fa640 20 reqs_thread_entry: start 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2bbce11800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2bbce11800 20 Filter name: none 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2ae6ffd640 20 reqs_thread_entry: start 2023-12-03T11:56:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.079+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.080+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:23.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.080+0000 7f2bbce11800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.080+0000 7f2bbce11800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:23.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.080+0000 7f2bbce11800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:23.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.081+0000 7f2b7e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.083+0000 7f2bbce11800 20 remove_watcher() i=0 2023-12-03T11:56:23.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.083+0000 7f2bbce11800 2 removed watcher, disabling cache 2023-12-03T11:56:23.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.084+0000 7f2bbce11800 20 remove_watcher() i=1 2023-12-03T11:56:23.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.085+0000 7f2bbce11800 20 remove_watcher() i=2 2023-12-03T11:56:23.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.086+0000 7f2bbce11800 20 remove_watcher() i=3 2023-12-03T11:56:23.088 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.087+0000 7f2bbce11800 20 remove_watcher() i=4 2023-12-03T11:56:23.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.088+0000 7f2bbce11800 20 remove_watcher() i=5 2023-12-03T11:56:23.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.089+0000 7f2bbce11800 20 remove_watcher() i=6 2023-12-03T11:56:23.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.090+0000 7f2bbce11800 20 remove_watcher() i=7 2023-12-03T11:56:23.097 INFO:tasks.workunit.client.0.smithi116.stderr:rados -p default.rgw.buckets.index setomapval .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 --omap-key-file /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_key < /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_val 2023-12-03T11:56:23.121 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.122 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.137 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket a-bucket 2023-12-03T11:56:23.164 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.164 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.177 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.176+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 realm 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.178+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.179+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.179+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.179+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.181 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.179+0000 7f5e20a8a800 20 searching for the correct realm 2023-12-03T11:56:23.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.189+0000 7f5e1e58a640 0 -- 172.21.15.116:0/3450313057 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55fcd31a55b0 msgr2=0x55fcd31a79a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.190+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 zone default found 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 4 Realm: () 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 zonegroup default 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.191+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.192+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.192+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.192+0000 7f5e20a8a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.192+0000 7f5e20a8a800 20 started sync module instance, tier type = 2023-12-03T11:56:23.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.192+0000 7f5e20a8a800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.199 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.198+0000 7f5e1ed8b640 0 -- 172.21.15.116:0/3450313057 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f5e10057970 msgr2=0x7f5e10058990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=0 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=1 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=2 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=3 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=4 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=5 2023-12-03T11:56:23.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=6 2023-12-03T11:56:23.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 20 add_watcher() i=7 2023-12-03T11:56:23.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.201+0000 7f5e20a8a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.203+0000 7f5de3fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.203+0000 7f5e20a8a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.204 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.203+0000 7f5e20a8a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5e20a8a800 5 note: GC not initialized 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5d52ffd640 20 reqs_thread_entry: start 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5e20a8a800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5e20a8a800 20 Filter name: none 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5d50ff9640 20 reqs_thread_entry: start 2023-12-03T11:56:23.211 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.209+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=13 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 INFO: main: starting shard=0 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:56:23.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.210+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: starting shard=1 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.213 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: starting shard=2 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 INFO: main: starting shard=3 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.211+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 INFO: main: starting shard=4 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:56:23.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 INFO: main: starting shard=5 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.212+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: starting shard=6 2023-12-03T11:56:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: starting shard=7 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 20 INFO: main: starting shard=8 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.213+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:56:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 INFO: main: starting shard=9 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 INFO: main: starting shard=10 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.214+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: starting shard=11 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:56:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:56:23.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: starting shard=12 2023-12-03T11:56:23.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:56:23.219 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.215+0000 7f5e20a8a800 20 INFO: main: done 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.216+0000 7f5de3fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.218+0000 7f5e20a8a800 20 remove_watcher() i=0 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.218+0000 7f5e20a8a800 2 removed watcher, disabling cache 2023-12-03T11:56:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.219+0000 7f5e20a8a800 20 remove_watcher() i=1 2023-12-03T11:56:23.221 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.220+0000 7f5e20a8a800 20 remove_watcher() i=2 2023-12-03T11:56:23.222 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.221+0000 7f5e20a8a800 20 remove_watcher() i=3 2023-12-03T11:56:23.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.222+0000 7f5e20a8a800 20 remove_watcher() i=4 2023-12-03T11:56:23.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.223+0000 7f5e20a8a800 20 remove_watcher() i=5 2023-12-03T11:56:23.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.224+0000 7f5e20a8a800 20 remove_watcher() i=6 2023-12-03T11:56:23.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.225+0000 7f5e20a8a800 20 remove_watcher() i=7 2023-12-03T11:56:23.232 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket a-bucket --num-shards 1 --yes-i-really-mean-it 2023-12-03T11:56:23.262 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.262 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.274+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 realm 2023-12-03T11:56:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.276+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.277+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.277+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.277+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.277+0000 7fd01af28800 20 searching for the correct realm 2023-12-03T11:56:23.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.286+0000 7fd016b77640 0 -- 172.21.15.116:0/482502953 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55c61eebd110 msgr2=0x55c61eebf500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.287+0000 7fd01af28800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.287+0000 7fd01af28800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.287+0000 7fd01af28800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.287+0000 7fd01af28800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.287+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 zone default found 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 4 Realm: () 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 zonegroup default 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.288+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.289+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.289+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.289+0000 7fd01af28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.289+0000 7fd01af28800 20 started sync module instance, tier type = 2023-12-03T11:56:23.292 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.289+0000 7fd01af28800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.294+0000 7fd017378640 0 -- 172.21.15.116:0/482502953 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7fd00c07a850 msgr2=0x7fd00c06e0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=0 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=1 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=2 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=3 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=4 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.297+0000 7fd01af28800 20 add_watcher() i=5 2023-12-03T11:56:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.298+0000 7fd01af28800 20 add_watcher() i=6 2023-12-03T11:56:23.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.299+0000 7fd01af28800 20 add_watcher() i=7 2023-12-03T11:56:23.300 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.299+0000 7fd01af28800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.300+0000 7fcf57fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.300+0000 7fd01af28800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.300+0000 7fd01af28800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.306+0000 7fd01af28800 5 note: GC not initialized 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.306+0000 7fcf4effd640 20 reqs_thread_entry: start 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.306+0000 7fd01af28800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.306+0000 7fd01af28800 20 Filter name: none 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fcf4cff9640 20 reqs_thread_entry: start 2023-12-03T11:56:23.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:56:23.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : miss 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.307+0000 7fd01af28800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.311+0000 7fd01af28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.311+0000 7fd01af28800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.311+0000 7fd01af28800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.311+0000 7fd01af28800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fcffcff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660551 cookie 94309410692208 notifier 7100 bl.length()=789 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fcffcff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fcffcff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:56:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fcffcff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:56:23.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:56:23.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:56:23.314 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.312+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:56:23.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.3.0 2023-12-03T11:56:23.316 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.3.0 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.0 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 10 Dropping entry with empty name, idx=�1001_04/57/0457f727ec113e418d5b16d206b200ed068c0533554883ce811df7c932a3df68/2018_12_11/2889999/3386469/metadata.gz 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.1 2023-12-03T11:56:23.317 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.315+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.2 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.316+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.316+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.3 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.316+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.316+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.4 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:56:23.318 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.5 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.6 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.317+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.7 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.318+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:56:23.319 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.318+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.8 2023-12-03T11:56:23.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.319+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:56:23.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.319+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.9 2023-12-03T11:56:23.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.319+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:56:23.320 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.319+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.10 2023-12-03T11:56:23.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:56:23.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.11 2023-12-03T11:56:23.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:56:23.321 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.2.12 2023-12-03T11:56:23.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:56:23.322 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.320+0000 7fd01af28800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 2023-12-03T11:56:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:56:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:56:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16, ofs0, ns] 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fcffcff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660552 cookie 94309410692208 notifier 7100 bl.length()=757 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fcffcff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 info.flags=0x17 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fcffcff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16 to cache LRU end 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fcffcff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_sources= new_sources= 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential sources added=[] removed=[] 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): orig_dests= new_dests= 2023-12-03T11:56:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.322+0000 7fd01af28800 20 handle_bi_update(): bucket=:a-bucket[21f2df23-11b5-4c91-821f-a8e55156195e.4668.16]): potential dests added=[] removed=[] 2023-12-03T11:56:23.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.325+0000 7fd01af28800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:56:23.328 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.327+0000 7fcf57fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.329+0000 7fd01af28800 20 remove_watcher() i=0 2023-12-03T11:56:23.330 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.329+0000 7fd01af28800 2 removed watcher, disabling cache 2023-12-03T11:56:23.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.330+0000 7fd01af28800 20 remove_watcher() i=1 2023-12-03T11:56:23.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.330+0000 7fd01af28800 20 remove_watcher() i=2 2023-12-03T11:56:23.333 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.332+0000 7fd01af28800 20 remove_watcher() i=3 2023-12-03T11:56:23.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.332+0000 7fd01af28800 20 remove_watcher() i=4 2023-12-03T11:56:23.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.333+0000 7fd01af28800 20 remove_watcher() i=5 2023-12-03T11:56:23.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.334+0000 7fd01af28800 20 remove_watcher() i=6 2023-12-03T11:56:23.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.335+0000 7fd01af28800 20 remove_watcher() i=7 2023-12-03T11:56:23.343 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket a-bucket 2023-12-03T11:56:23.370 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.370 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.382+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 20 realm 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.383+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.384+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.384+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.384+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.384+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.386 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.385+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.385+0000 7f91ed5a9800 20 searching for the correct realm 2023-12-03T11:56:23.394 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.393+0000 7f91ea8a8640 0 -- 172.21.15.116:0/2705495913 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x55d67186c610 msgr2=0x55d67186ea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.395+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 zone default found 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 4 Realm: () 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 zonegroup default 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.396+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.397+0000 7f91ed5a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.397+0000 7f91ed5a9800 20 started sync module instance, tier type = 2023-12-03T11:56:23.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.397+0000 7f91ed5a9800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.400+0000 7f91eb8aa640 0 -- 172.21.15.116:0/2705495913 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7f91e0056b10 msgr2=0x7f91e005b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=0 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=1 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=2 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=3 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=4 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=5 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.405+0000 7f91ed5a9800 20 add_watcher() i=6 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.406+0000 7f91ed5a9800 20 add_watcher() i=7 2023-12-03T11:56:23.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.406+0000 7f91ed5a9800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.408+0000 7f91b17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.408+0000 7f91ed5a9800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.409 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.408+0000 7f91ed5a9800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.415 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f91ed5a9800 5 note: GC not initialized 2023-12-03T11:56:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f911ffff640 20 reqs_thread_entry: start 2023-12-03T11:56:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f91ed5a9800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f91ed5a9800 20 Filter name: none 2023-12-03T11:56:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f911dffb640 20 reqs_thread_entry: start 2023-12-03T11:56:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.414+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 INFO: main: max_entries=1000, index=gen=3, max_shards=1 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 INFO: main: starting shard=0 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.3.0 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.415+0000 7f91ed5a9800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.16.3.0 2023-12-03T11:56:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.416+0000 7f91ed5a9800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:23.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.416+0000 7f91ed5a9800 20 INFO: main: done 2023-12-03T11:56:23.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.416+0000 7f91b17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.418+0000 7f91ed5a9800 20 remove_watcher() i=0 2023-12-03T11:56:23.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.418+0000 7f91ed5a9800 2 removed watcher, disabling cache 2023-12-03T11:56:23.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.420+0000 7f91ed5a9800 20 remove_watcher() i=1 2023-12-03T11:56:23.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.421+0000 7f91ed5a9800 20 remove_watcher() i=2 2023-12-03T11:56:23.423 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.422+0000 7f91ed5a9800 20 remove_watcher() i=3 2023-12-03T11:56:23.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.422+0000 7f91ed5a9800 20 remove_watcher() i=4 2023-12-03T11:56:23.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.424+0000 7f91ed5a9800 20 remove_watcher() i=5 2023-12-03T11:56:23.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.425+0000 7f91ed5a9800 20 remove_watcher() i=6 2023-12-03T11:56:23.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.425+0000 7f91ed5a9800 20 remove_watcher() i=7 2023-12-03T11:56:23.431 INFO:tasks.workunit.client.0.smithi116.stderr: test: PUT succeeds during reshard 2023-12-03T11:56:23.433 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:56:23.460 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.460 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.472+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.473+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.475 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.473+0000 7f2cec325800 20 realm 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.473+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.474+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.474+0000 7f2cec325800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.474+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.474+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.474+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.475+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.475+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.475+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.475+0000 7f2cec325800 20 searching for the correct realm 2023-12-03T11:56:23.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.484+0000 7f2ce7f4d640 0 -- 172.21.15.116:0/1854397512 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2ce0017070 msgr2=0x7f2ce0032950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.486 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.485+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 zone default found 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 4 Realm: () 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 zonegroup default 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.486+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.487+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.487+0000 7f2cec325800 20 started sync module instance, tier type = 2023-12-03T11:56:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.487+0000 7f2cec325800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.493 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.492+0000 7f2ce7f4d640 0 -- 172.21.15.116:0/1854397512 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2cc0079e40 msgr2=0x7f2cc007c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=0 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=1 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=2 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=3 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=4 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=5 2023-12-03T11:56:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.495+0000 7f2cec325800 20 add_watcher() i=6 2023-12-03T11:56:23.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.496+0000 7f2cec325800 20 add_watcher() i=7 2023-12-03T11:56:23.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.496+0000 7f2cec325800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.498+0000 7f2cadfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.498+0000 7f2cec325800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.499 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.498+0000 7f2cec325800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2cec325800 5 note: GC not initialized 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2c24ff9640 20 reqs_thread_entry: start 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2cec325800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2cec325800 20 Filter name: none 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2c1a7fc640 20 reqs_thread_entry: start 2023-12-03T11:56:23.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.504+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:23.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.505+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.506+0000 7f2cec325800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:23.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.506+0000 7f2cec325800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:23.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.507+0000 7f2cec325800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.508 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.507+0000 7f2cec325800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.509 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.508+0000 7f2cadfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.510+0000 7f2cec325800 20 remove_watcher() i=0 2023-12-03T11:56:23.511 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.510+0000 7f2cec325800 2 removed watcher, disabling cache 2023-12-03T11:56:23.512 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.511+0000 7f2cec325800 20 remove_watcher() i=1 2023-12-03T11:56:23.513 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.512+0000 7f2cec325800 20 remove_watcher() i=2 2023-12-03T11:56:23.514 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.513+0000 7f2cec325800 20 remove_watcher() i=3 2023-12-03T11:56:23.515 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.514+0000 7f2cec325800 20 remove_watcher() i=4 2023-12-03T11:56:23.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.515+0000 7f2cec325800 20 remove_watcher() i=5 2023-12-03T11:56:23.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.516+0000 7f2cec325800 20 remove_watcher() i=6 2023-12-03T11:56:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.517+0000 7f2cec325800 20 remove_watcher() i=7 2023-12-03T11:56:23.523 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:23.523 INFO:tasks.workunit.client.0.smithi116.stderr:BUCKET_STATS: 2023-12-03T11:56:23.523 INFO:tasks.workunit.client.0.smithi116.stderr:bucket: myver id: 21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 num_objects: 0 size_kb: 0 num_shards: 12 2023-12-03T11:56:23.523 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:23.524 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin --inject-delay-at=do_reshard --inject-delay-ms=5000 bucket reshard --bucket myver --num-shards 13 2023-12-03T11:56:23.555 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:23.555 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:23.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.568+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.569+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.569+0000 7f073b233800 20 realm 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.569+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.570+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.571+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:23.572 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.571+0000 7f073b233800 20 searching for the correct realm 2023-12-03T11:56:23.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.779+0000 7f0738532640 0 -- 172.21.15.116:0/3137820864 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x55c8b7883180 msgr2=0x55c8b7885570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.782+0000 7f073b233800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.782+0000 7f073b233800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.782+0000 7f073b233800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.782+0000 7f073b233800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.782+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 zone default found 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 4 Realm: () 2023-12-03T11:56:23.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 zonegroup default 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.783+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f0738532640 0 -- 172.21.15.116:0/3137820864 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55c8b788bf50 msgr2=0x55c8b788e340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f073b233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f073b233800 20 started sync module instance, tier type = 2023-12-03T11:56:23.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.784+0000 7f073b233800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=0 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=1 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=2 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=3 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=4 2023-12-03T11:56:23.794 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=5 2023-12-03T11:56:23.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.792+0000 7f073b233800 20 add_watcher() i=6 2023-12-03T11:56:23.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.793+0000 7f073b233800 20 add_watcher() i=7 2023-12-03T11:56:23.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.793+0000 7f073b233800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:23.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.795+0000 7f06fefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:23.796 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.795+0000 7f073b233800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:23.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.795+0000 7f073b233800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:23.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.801+0000 7f073b233800 5 note: GC not initialized 2023-12-03T11:56:23.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.801+0000 7f066dffb640 20 reqs_thread_entry: start 2023-12-03T11:56:23.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.801+0000 7f073b233800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:23.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.801+0000 7f073b233800 20 Filter name: none 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f065f7fe640 20 reqs_thread_entry: start 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : miss 2023-12-03T11:56:23.804 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 20 rados->read ofs=0 len=0 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 adding default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.802+0000 7f073b233800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:23.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.808+0000 7f073b233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:23.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.808+0000 7f073b233800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:23.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.808+0000 7f073b233800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.808+0000 7f073b233800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f071f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660542 cookie 94320561031920 notifier 7602 bl.length()=783 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f071f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f071f7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f071f7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:23.811 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:23.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.809+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:23.813 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:23.812+0000 7f073b233800 -1 Injecting delay=5.000000000s at location=do_reshard 2023-12-03T11:56:24.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:24.528 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:28.814 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:28.815 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.0 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.812+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.0 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.813+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.1 2023-12-03T11:56:28.816 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.813+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.1 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.813+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.2 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.813+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.2 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.814+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.3 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.814+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.3 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.815+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.4 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.815+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.4 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.815+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.5 2023-12-03T11:56:28.817 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.815+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.5 2023-12-03T11:56:28.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.816+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.6 2023-12-03T11:56:28.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.816+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.6 2023-12-03T11:56:28.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.817+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.7 2023-12-03T11:56:28.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.817+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.7 2023-12-03T11:56:28.818 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.817+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.8 2023-12-03T11:56:28.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.817+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.8 2023-12-03T11:56:28.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.818+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.9 2023-12-03T11:56:28.819 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.818+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.9 2023-12-03T11:56:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.818+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.10 2023-12-03T11:56:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.818+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.10 2023-12-03T11:56:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.819+0000 7f073b233800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.11 2023-12-03T11:56:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.819+0000 7f073b233800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.1.11 2023-12-03T11:56:28.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.820+0000 7f073b233800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:56:28.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.820+0000 7f073b233800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:28.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.821+0000 7f073b233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:28.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.821+0000 7f073b233800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:28.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.821+0000 7f073b233800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:28.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.821+0000 7f073b233800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:28.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f071f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660543 cookie 94320561031920 notifier 7602 bl.length()=751 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f071f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f071f7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f071f7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:28.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.822+0000 7f073b233800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:28.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.825+0000 7f073b233800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:56:28.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.827+0000 7f06fefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:28.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.830+0000 7f073b233800 20 remove_watcher() i=0 2023-12-03T11:56:28.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.830+0000 7f073b233800 2 removed watcher, disabling cache 2023-12-03T11:56:28.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.832+0000 7f073b233800 20 remove_watcher() i=1 2023-12-03T11:56:28.835 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.833+0000 7f073b233800 20 remove_watcher() i=2 2023-12-03T11:56:28.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.835+0000 7f073b233800 20 remove_watcher() i=3 2023-12-03T11:56:28.837 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.836+0000 7f073b233800 20 remove_watcher() i=4 2023-12-03T11:56:28.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.837+0000 7f073b233800 20 remove_watcher() i=5 2023-12-03T11:56:28.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.839+0000 7f073b233800 20 remove_watcher() i=6 2023-12-03T11:56:28.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:28.840+0000 7f073b233800 20 remove_watcher() i=7 2023-12-03T11:56:29.555 INFO:tasks.workunit.client.0.smithi116.stderr:put object successful 2023-12-03T11:56:29.555 INFO:tasks.workunit.client.0.smithi116.stderr:TEST: check that bucket stats are correct after reshard with unlinked entries 2023-12-03T11:56:29.555 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-12-03T11:56:29.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.557 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.603 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.646 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.663 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.667 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.682 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.685 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:29.700 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:29.704 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:56:30.083 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:56:31.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.461 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.516 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:56:31.518 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket myver --object x 2023-12-03T11:56:31.550 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:31.550 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:31.565 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.564+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 realm 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.568 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.566+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.567+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.567+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.567+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:31.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.567+0000 7fb309327800 20 searching for the correct realm 2023-12-03T11:56:31.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.577+0000 7fb3058c0640 0 -- 172.21.15.116:0/2504106201 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x7fb2e0079e10 msgr2=0x7fb2e007c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.581 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.582 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.580+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.583 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 zone default found 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 4 Realm: () 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 zonegroup default 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.581+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.582+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.582+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.582+0000 7fb309327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.582+0000 7fb309327800 20 started sync module instance, tier type = 2023-12-03T11:56:31.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.582+0000 7fb309327800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:31.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.586+0000 7fb3058c0640 0 -- 172.21.15.116:0/2504106201 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7fb2f0057eb0 msgr2=0x7fb2f005d7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=0 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=1 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=2 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=3 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=4 2023-12-03T11:56:31.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=5 2023-12-03T11:56:31.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.592+0000 7fb309327800 20 add_watcher() i=6 2023-12-03T11:56:31.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.593+0000 7fb309327800 20 add_watcher() i=7 2023-12-03T11:56:31.594 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.593+0000 7fb309327800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:31.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.595+0000 7fb2cafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.595+0000 7fb309327800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:31.596 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.595+0000 7fb309327800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.601+0000 7fb309327800 5 note: GC not initialized 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.601+0000 7fb239ffb640 20 reqs_thread_entry: start 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.601+0000 7fb309327800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.601+0000 7fb309327800 20 Filter name: none 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.602+0000 7fb22b7fe640 20 reqs_thread_entry: start 2023-12-03T11:56:31.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.602+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.602+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.602+0000 7fb309327800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=13 2023-12-03T11:56:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 INFO: main: starting shard=0 2023-12-03T11:56:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 INFO: main: starting shard=1 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.603+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.604+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.604+0000 7fb309327800 20 INFO: main: starting shard=2 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.604+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.604+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 INFO: main: starting shard=3 2023-12-03T11:56:31.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 INFO: main: starting shard=4 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.605+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 INFO: main: starting shard=5 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 INFO: main: starting shard=6 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.606+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 INFO: main: starting shard=7 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 INFO: main: starting shard=8 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.607+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 INFO: main: starting shard=9 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 INFO: main: starting shard=10 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.608+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.609+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.609+0000 7fb309327800 20 INFO: main: starting shard=11 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.609+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.609+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 20 INFO: main: starting shard=12 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.610+0000 7fb309327800 20 INFO: main: done 2023-12-03T11:56:31.612 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.611+0000 7fb2cafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.613+0000 7fb309327800 20 remove_watcher() i=0 2023-12-03T11:56:31.615 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.613+0000 7fb309327800 2 removed watcher, disabling cache 2023-12-03T11:56:31.616 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.614+0000 7fb309327800 20 remove_watcher() i=1 2023-12-03T11:56:31.617 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.615+0000 7fb309327800 20 remove_watcher() i=2 2023-12-03T11:56:31.618 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.616+0000 7fb309327800 20 remove_watcher() i=3 2023-12-03T11:56:31.619 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.618+0000 7fb309327800 20 remove_watcher() i=4 2023-12-03T11:56:31.620 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.619+0000 7fb309327800 20 remove_watcher() i=5 2023-12-03T11:56:31.621 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.620+0000 7fb309327800 20 remove_watcher() i=6 2023-12-03T11:56:31.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.620+0000 7fb309327800 20 remove_watcher() i=7 2023-12-03T11:56:31.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.629 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.633 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.683 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:56:31.685 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket myver --object y 2023-12-03T11:56:31.715 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:31.715 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:31.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.727+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.730 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 realm 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.731 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.729+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.730+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.730+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.730+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.730+0000 7f66f9a13800 20 searching for the correct realm 2023-12-03T11:56:31.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.740+0000 7f66f7513640 0 -- 172.21.15.116:0/3355212814 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x55ce5c1a1d70 msgr2=0x55ce5c1a4160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.741+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:31.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 zone default found 2023-12-03T11:56:31.744 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 4 Realm: () 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 zonegroup default 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 started sync module instance, tier type = 2023-12-03T11:56:31.745 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.742+0000 7f66f9a13800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:31.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.750+0000 7f66eed12640 0 -- 172.21.15.116:0/3355212814 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x55ce5c1a6ca0 msgr2=0x55ce5c1a9090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=0 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=1 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=2 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=3 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=4 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=5 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.751+0000 7f66f9a13800 20 add_watcher() i=6 2023-12-03T11:56:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.752+0000 7f66f9a13800 20 add_watcher() i=7 2023-12-03T11:56:31.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.752+0000 7f66f9a13800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:31.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.753+0000 7f66bd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.754+0000 7f66f9a13800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:31.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.754+0000 7f66f9a13800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.759+0000 7f66f9a13800 5 note: GC not initialized 2023-12-03T11:56:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f662bfff640 20 reqs_thread_entry: start 2023-12-03T11:56:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f66f9a13800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f66f9a13800 20 Filter name: none 2023-12-03T11:56:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f6629ffb640 20 reqs_thread_entry: start 2023-12-03T11:56:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.760+0000 7f66f9a13800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=13 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 INFO: main: starting shard=0 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 INFO: main: starting shard=1 2023-12-03T11:56:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.761+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: starting shard=2 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: starting shard=3 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 INFO: main: starting shard=4 2023-12-03T11:56:31.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.762+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 INFO: main: starting shard=5 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 INFO: main: starting shard=6 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.766 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.763+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 INFO: main: starting shard=7 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 INFO: main: starting shard=8 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.764+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 INFO: main: starting shard=9 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 INFO: main: starting shard=10 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.765+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 INFO: main: starting shard=11 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 INFO: main: starting shard=12 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.766+0000 7f66f9a13800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.767+0000 7f66f9a13800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.767+0000 7f66f9a13800 20 INFO: main: done 2023-12-03T11:56:31.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.767+0000 7f66bd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.769+0000 7f66f9a13800 20 remove_watcher() i=0 2023-12-03T11:56:31.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.769+0000 7f66f9a13800 2 removed watcher, disabling cache 2023-12-03T11:56:31.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.770+0000 7f66f9a13800 20 remove_watcher() i=1 2023-12-03T11:56:31.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.771+0000 7f66f9a13800 20 remove_watcher() i=2 2023-12-03T11:56:31.774 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.773+0000 7f66f9a13800 20 remove_watcher() i=3 2023-12-03T11:56:31.775 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.773+0000 7f66f9a13800 20 remove_watcher() i=4 2023-12-03T11:56:31.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.774+0000 7f66f9a13800 20 remove_watcher() i=5 2023-12-03T11:56:31.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.775+0000 7f66f9a13800 20 remove_watcher() i=6 2023-12-03T11:56:31.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.776+0000 7f66f9a13800 20 remove_watcher() i=7 2023-12-03T11:56:31.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.785 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.788 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.839 INFO:tasks.workunit.client.0.smithi116.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:56:31.840 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bi list --bucket myver --object z 2023-12-03T11:56:31.870 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:31.870 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:31.883 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.882+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.883+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.883+0000 7fe4ccb21800 20 realm 2023-12-03T11:56:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.883+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.884+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.884+0000 7fe4ccb21800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.884+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.884+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.884+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.885+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.885+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.885+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:31.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.885+0000 7fe4ccb21800 20 searching for the correct realm 2023-12-03T11:56:31.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.895+0000 7fe4c80e4640 0 -- 172.21.15.116:0/2641652931 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x5640d22670b0 msgr2=0x5640d22694a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.896+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 zone default found 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 4 Realm: () 2023-12-03T11:56:31.899 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 zonegroup default 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.897+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.898+0000 7fe4ccb21800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.898+0000 7fe4ccb21800 20 started sync module instance, tier type = 2023-12-03T11:56:31.900 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.898+0000 7fe4ccb21800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.904+0000 7fe4c80e4640 0 -- 172.21.15.116:0/2641652931 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7fe4a0079e40 msgr2=0x7fe4a007c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=0 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=1 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=2 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=3 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=4 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=5 2023-12-03T11:56:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=6 2023-12-03T11:56:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 20 add_watcher() i=7 2023-12-03T11:56:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.907+0000 7fe4ccb21800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.909+0000 7fe48f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.910+0000 7fe4ccb21800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.910+0000 7fe4ccb21800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.915+0000 7fe4ccb21800 5 note: GC not initialized 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.915+0000 7fe3fe7fc640 20 reqs_thread_entry: start 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe4ccb21800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe4ccb21800 20 Filter name: none 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe3effff640 20 reqs_thread_entry: start 2023-12-03T11:56:31.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:31.918 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.916+0000 7fe4ccb21800 20 rados->read ofs=0 len=0 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=13 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 INFO: main: starting shard=0 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 INFO: main: starting shard=1 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.919 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.917+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: starting shard=2 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: starting shard=3 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 INFO: main: starting shard=4 2023-12-03T11:56:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.918+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 INFO: main: starting shard=5 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 INFO: main: starting shard=6 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.919+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 INFO: main: starting shard=7 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.922 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 INFO: main: starting shard=8 2023-12-03T11:56:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.920+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 INFO: main: starting shard=9 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 INFO: main: starting shard=10 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.924 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.921+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 INFO: main: starting shard=11 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 INFO: main: starting shard=12 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.922+0000 7fe4ccb21800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.923+0000 7fe4ccb21800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:56:31.925 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.923+0000 7fe4ccb21800 20 INFO: main: done 2023-12-03T11:56:31.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.923+0000 7fe48f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.925+0000 7fe4ccb21800 20 remove_watcher() i=0 2023-12-03T11:56:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.925+0000 7fe4ccb21800 2 removed watcher, disabling cache 2023-12-03T11:56:31.928 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.926+0000 7fe4ccb21800 20 remove_watcher() i=1 2023-12-03T11:56:31.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.927+0000 7fe4ccb21800 20 remove_watcher() i=2 2023-12-03T11:56:31.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.929+0000 7fe4ccb21800 20 remove_watcher() i=3 2023-12-03T11:56:31.931 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.929+0000 7fe4ccb21800 20 remove_watcher() i=4 2023-12-03T11:56:31.932 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.930+0000 7fe4ccb21800 20 remove_watcher() i=5 2023-12-03T11:56:31.933 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.931+0000 7fe4ccb21800 20 remove_watcher() i=6 2023-12-03T11:56:31.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:31.932+0000 7fe4ccb21800 20 remove_watcher() i=7 2023-12-03T11:56:31.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:31.941 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:31.944 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:56:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:56:32.694 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 17 --yes-i-really-mean-it 2023-12-03T11:56:32.723 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:32.723 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:32.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.735+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 realm 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.737+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.738+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:32.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.738+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.738+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:32.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.738+0000 7fd349122800 20 searching for the correct realm 2023-12-03T11:56:32.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.747+0000 7fd3455a7640 0 -- 172.21.15.116:0/1997050766 >> [v2:172.21.15.116:6800/3848060595,v1:172.21.15.116:6801/3848060595] conn(0x559ecefebfe0 msgr2=0x559ecefee3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:32.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:32.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.749+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.754 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 zone default found 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 4 Realm: () 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:32.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 zonegroup default 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.750+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.751+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.751+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.751+0000 7fd349122800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.751+0000 7fd349122800 20 started sync module instance, tier type = 2023-12-03T11:56:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.751+0000 7fd349122800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:32.757 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.755+0000 7fd3455a7640 0 -- 172.21.15.116:0/1997050766 >> [v2:172.21.15.152:6816/1133785446,v1:172.21.15.152:6818/1133785446] conn(0x7fd32c056fb0 msgr2=0x7fd32c06aaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:56:32.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=0 2023-12-03T11:56:32.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=1 2023-12-03T11:56:32.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=2 2023-12-03T11:56:32.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=3 2023-12-03T11:56:32.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=4 2023-12-03T11:56:32.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=5 2023-12-03T11:56:32.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.760+0000 7fd349122800 20 add_watcher() i=6 2023-12-03T11:56:32.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.761+0000 7fd349122800 20 add_watcher() i=7 2023-12-03T11:56:32.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.761+0000 7fd349122800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:32.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.763+0000 7fd30b7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:32.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.764+0000 7fd349122800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:32.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.764+0000 7fd349122800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.769+0000 7fd349122800 5 note: GC not initialized 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.769+0000 7fd27a7fc640 20 reqs_thread_entry: start 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.769+0000 7fd349122800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.769+0000 7fd349122800 20 Filter name: none 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd26bfff640 20 reqs_thread_entry: start 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:56:32.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:32.772 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : miss 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 adding default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:32.773 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.770+0000 7fd349122800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:32.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.776+0000 7fd349122800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:32.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.776+0000 7fd349122800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:32.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.776+0000 7fd349122800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:32.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.776+0000 7fd349122800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:32.778 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd33b5a3640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660544 cookie 94140566053984 notifier 7259 bl.length()=783 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd33b5a3640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd33b5a3640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd33b5a3640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:32.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.777+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:32.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.0 2023-12-03T11:56:32.781 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.0 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.1 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.1 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.2 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.2 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.3 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.3 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.4 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.4 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.5 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.5 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.6 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.6 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.7 2023-12-03T11:56:32.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.7 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.8 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.8 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.10 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.10 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.11 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.11 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.12 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.12 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.13 2023-12-03T11:56:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.13 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.14 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.14 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.15 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.15 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.16 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.16 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.779+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.0 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.780+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.780+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.1 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.780+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.780+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.2 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.781+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.781+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.3 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.781+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.781+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.4 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.782+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.782+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.5 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.782+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.782+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.6 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.783+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.783+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.7 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.783+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.783+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.8 2023-12-03T11:56:32.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.784+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.784+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.9 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.784+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.784+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.10 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.785+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.785+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.11 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.786+0000 7fd349122800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:32.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.786+0000 7fd349122800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.2.12 2023-12-03T11:56:32.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.787+0000 7fd349122800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:56:32.789 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.788+0000 7fd349122800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 10 moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17, ofs0, ns] 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd33ada2640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660545 cookie 94140566053984 notifier 7259 bl.length()=751 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd33ada2640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd33ada2640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd33ada2640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_sources= new_sources= 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential sources added=[] removed=[] 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): orig_dests= new_dests= 2023-12-03T11:56:32.791 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.789+0000 7fd349122800 20 handle_bi_update(): bucket=:myver[21f2df23-11b5-4c91-821f-a8e55156195e.4668.17]): potential dests added=[] removed=[] 2023-12-03T11:56:32.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.792+0000 7fd349122800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:56:32.795 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.793+0000 7fd30b7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.795+0000 7fd349122800 20 remove_watcher() i=0 2023-12-03T11:56:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.795+0000 7fd349122800 2 removed watcher, disabling cache 2023-12-03T11:56:32.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.796+0000 7fd349122800 20 remove_watcher() i=1 2023-12-03T11:56:32.799 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.797+0000 7fd349122800 20 remove_watcher() i=2 2023-12-03T11:56:32.800 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.798+0000 7fd349122800 20 remove_watcher() i=3 2023-12-03T11:56:32.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.799+0000 7fd349122800 20 remove_watcher() i=4 2023-12-03T11:56:32.801 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.800+0000 7fd349122800 20 remove_watcher() i=5 2023-12-03T11:56:32.802 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.801+0000 7fd349122800 20 remove_watcher() i=6 2023-12-03T11:56:32.803 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.802+0000 7fd349122800 20 remove_watcher() i=7 2023-12-03T11:56:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket check unlinked --bucket myver --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:56:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:32.842 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:32.857 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.855+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 realm 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:32.859 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.857+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.858+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.858+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.858+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:32.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.858+0000 7f37a1937800 20 searching for the correct realm 2023-12-03T11:56:32.867 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:32.866+0000 7f379d6cd640 0 -- 172.21.15.116:0/2687611006 >> [v2:172.21.15.116:6808/1519417773,v1:172.21.15.116:6809/1519417773] conn(0x7f3788029920 msgr2=0x7f3788028400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.069+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 zone default found 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 4 Realm: () 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 zonegroup default 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.070+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f3794ecc640 0 -- 172.21.15.116:0/2687611006 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x563acd74a9f0 msgr2=0x563acd74cde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:33.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f37a1937800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f37a1937800 20 started sync module instance, tier type = 2023-12-03T11:56:33.074 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.071+0000 7f37a1937800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=0 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=1 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=2 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=3 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=4 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=5 2023-12-03T11:56:33.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=6 2023-12-03T11:56:33.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 20 add_watcher() i=7 2023-12-03T11:56:33.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.080+0000 7f37a1937800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:33.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.082+0000 7f3762fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:33.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.083+0000 7f37a1937800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:33.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.083+0000 7f37a1937800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.088+0000 7f37a1937800 5 note: GC not initialized 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.088+0000 7f36d1ffb640 20 reqs_thread_entry: start 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 20 Filter name: none 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f36c37fe640 20 reqs_thread_entry: start 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 : miss 2023-12-03T11:56:33.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.089+0000 7f37a1937800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 info.flags=0x17 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 adding default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 to cache LRU end 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.0 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.0 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 1 NOTICE: finished shard 0 (0 entries removed) 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.1 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.090+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.1 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 1 NOTICE: finished shard 1 (0 entries removed) 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.2 2023-12-03T11:56:33.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.2 2023-12-03T11:56:33.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 1 NOTICE: finished shard 2 (0 entries removed) 2023-12-03T11:56:33.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.3 2023-12-03T11:56:33.094 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.091+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.3 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 1 NOTICE: finished shard 3 (0 entries removed) 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.4 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.4 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 1 NOTICE: finished shard 4 (0 entries removed) 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.5 2023-12-03T11:56:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.5 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 1 NOTICE: finished shard 5 (0 entries removed) 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.6 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.092+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.6 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 1 NOTICE: finished shard 6 (0 entries removed) 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.7 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.7 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 1 NOTICE: finished shard 7 (0 entries removed) 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.8 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.093+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.8 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.094+0000 7f37a1937800 1 NOTICE: finished shard 8 (0 entries removed) 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.094+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.094+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.094+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.10 2023-12-03T11:56:33.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.094+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.10 2023-12-03T11:56:33.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 1 NOTICE: finished shard 10 (0 entries removed) 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.11 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.11 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 1 NOTICE: finished shard 11 (0 entries removed) 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.12 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.095+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.12 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 1 NOTICE: finished shard 12 (0 entries removed) 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.13 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.13 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 1 NOTICE: finished shard 13 (0 entries removed) 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.14 2023-12-03T11:56:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.14 2023-12-03T11:56:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 1 NOTICE: finished shard 14 (0 entries removed) 2023-12-03T11:56:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.15 2023-12-03T11:56:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.096+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.15 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 1 NOTICE: finished shard 15 (0 entries removed) 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 10 bucket_obj is .dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.16 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.16 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 1 NOTICE: finished shard 16 (0 entries removed) 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.097+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:x state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.098+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to rzcjogw17xsgrpel204nc6byw41zbm4g 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.098+0000 7f37a1937800 20 get_obj_state_impl: setting s->olh_tag to druvvhb7gfs07xopa442azmoy3i80ied 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.098+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=druvvhb7gfs07xopa442azmoy3i80ied 2023-12-03T11:56:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.099+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.100+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.100+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=2 key=x[] 2023-12-03T11:56:33.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.100+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=3 key=x[NY7AJceskGX8Yrqpwcn2BuO0ftSWOCa] 2023-12-03T11:56:33.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:NY7AJceskGX8Yrqpwcn2BuO0ftSWOCa_x state=0x563acd806f98 s->prefetch_data=0 2023-12-03T11:56:33.102 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 10 manifest: total_size = 22 2023-12-03T11:56:33.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.12147331212824976165 2023-12-03T11:56:33.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:NY7AJceskGX8Yrqpwcn2BuO0ftSWOCa_x state=0x563acd806f98 s->prefetch_data=0 2023-12-03T11:56:33.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 20 prepare_atomic_modification: state is not atomic. state=0x563acd806f98 2023-12-03T11:56:33.103 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.101+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.104+0000 7f37a1937800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:56:33.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.104+0000 7f37a1937800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:56:33.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.104+0000 7f37a1937800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:56:33.105 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.104+0000 7f37a1937800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:NY7AJceskGX8Yrqpwcn2BuO0ftSWOCa_.eLaFbWc5DCT2c1dgaMPUZGjF06QnE43_0 2023-12-03T11:56:33.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.105+0000 7f377f7fe640 20 handle_completion(): completion ok for obj=x[NY7AJceskGX8Yrqpwcn2BuO0ftSWOCa] 2023-12-03T11:56:33.106 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.105+0000 7f37a1937800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:56:33.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.107+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:x state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.108 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.107+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=druvvhb7gfs07xopa442azmoy3i80ied 2023-12-03T11:56:33.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.111+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.113+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.113+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.113+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:y state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.114+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to c8zd4okjqk1lsbu2p4co53j1l5kababc 2023-12-03T11:56:33.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.114+0000 7f37a1937800 20 get_obj_state_impl: setting s->olh_tag to 62w5yn8nf83tokhzct5vplx4v6wwepr4 2023-12-03T11:56:33.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.114+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=62w5yn8nf83tokhzct5vplx4v6wwepr4 2023-12-03T11:56:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.115+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.116+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.116+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=2 key=y[] 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.116+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=3 key=y[B-w9PlWOhhWiVa6qgH1e.MLVh1iF9cY] 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.116+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:B-w9PlWOhhWiVa6qgH1e.MLVh1iF9cY_y state=0x563acd802f58 s->prefetch_data=0 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.117+0000 7f37a1937800 10 manifest: total_size = 22 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.117+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.5834479330611942026 2023-12-03T11:56:33.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.117+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:B-w9PlWOhhWiVa6qgH1e.MLVh1iF9cY_y state=0x563acd802f58 s->prefetch_data=0 2023-12-03T11:56:33.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.117+0000 7f37a1937800 20 prepare_atomic_modification: state is not atomic. state=0x563acd802f58 2023-12-03T11:56:33.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.117+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.119+0000 7f37a1937800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:56:33.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.119+0000 7f37a1937800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:56:33.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.119+0000 7f37a1937800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:56:33.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.119+0000 7f37a1937800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:B-w9PlWOhhWiVa6qgH1e.MLVh1iF9cY_.6XLLZ5ecwOn_Yw5z0QcYnF6oGs17SAm_0 2023-12-03T11:56:33.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.121+0000 7f377effd640 20 handle_completion(): completion ok for obj=y[B-w9PlWOhhWiVa6qgH1e.MLVh1iF9cY] 2023-12-03T11:56:33.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.121+0000 7f37a1937800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:56:33.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.122+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:y state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.124 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.122+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=62w5yn8nf83tokhzct5vplx4v6wwepr4 2023-12-03T11:56:33.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.125+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.126+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.126+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.127+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:z state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.128 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.127+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to dxhdt53w8itydw8ko64aj2n3i1mn8ocl 2023-12-03T11:56:33.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.127+0000 7f37a1937800 20 get_obj_state_impl: setting s->olh_tag to 0uz7hpc9r3rhhfypos1uh0w4ynaykxwx 2023-12-03T11:56:33.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.127+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0uz7hpc9r3rhhfypos1uh0w4ynaykxwx 2023-12-03T11:56:33.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.128+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=2 key=z[] 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 olh_log_entry: epoch=2 op=3 key=z[0u-MdE2tVyRws9b902BowqKRFMmpdQU] 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:0u-MdE2tVyRws9b902BowqKRFMmpdQU_z state=0x563acd801f48 s->prefetch_data=0 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 10 manifest: total_size = 11 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 get_obj_state: setting s->obj_tag to 21f2df23-11b5-4c91-821f-a8e55156195e.4662.9922575820324831208 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:_:0u-MdE2tVyRws9b902BowqKRFMmpdQU_z state=0x563acd801f48 s->prefetch_data=0 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 prepare_atomic_modification: state is not atomic. state=0x563acd801f48 2023-12-03T11:56:33.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.130+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.133+0000 7f37a1937800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:56:33.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.133+0000 7f37a1937800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:56:33.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.133+0000 7f37a1937800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:56:33.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.133+0000 7f37a1937800 5 delete_objs_inline: removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:0u-MdE2tVyRws9b902BowqKRFMmpdQU_.1-vPfu98PDc8z9Ih7AzkHepbGqraa-Z_0 2023-12-03T11:56:33.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.134+0000 7f377f7fe640 20 handle_completion(): completion ok for obj=z[0u-MdE2tVyRws9b902BowqKRFMmpdQU] 2023-12-03T11:56:33.135 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.134+0000 7f37a1937800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:56:33.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.135+0000 7f37a1937800 20 get_obj_state: octx=0x7f3790089040 obj=myver:z state=0x563acd7fead8 s->prefetch_data=0 2023-12-03T11:56:33.137 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.135+0000 7f37a1937800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0uz7hpc9r3rhhfypos1uh0w4ynaykxwx 2023-12-03T11:56:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.138+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.139+0000 7f37a1937800 20 bucket index object: default.rgw.buckets.index:.dir.21f2df23-11b5-4c91-821f-a8e55156195e.4668.17.3.9 2023-12-03T11:56:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.139+0000 7f37a1937800 1 NOTICE: finished shard 9 (3 entries removed) 2023-12-03T11:56:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.139+0000 7f37a1937800 1 NOTICE: finished all shards (3 entries removed) 2023-12-03T11:56:33.141 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.140+0000 7f3762fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:33.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.141+0000 7f37a1937800 20 remove_watcher() i=0 2023-12-03T11:56:33.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.141+0000 7f37a1937800 2 removed watcher, disabling cache 2023-12-03T11:56:33.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.142+0000 7f37a1937800 20 remove_watcher() i=1 2023-12-03T11:56:33.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.143+0000 7f37a1937800 20 remove_watcher() i=2 2023-12-03T11:56:33.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.144+0000 7f37a1937800 20 remove_watcher() i=3 2023-12-03T11:56:33.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.145+0000 7f37a1937800 20 remove_watcher() i=4 2023-12-03T11:56:33.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.146+0000 7f37a1937800 20 remove_watcher() i=5 2023-12-03T11:56:33.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.147+0000 7f37a1937800 20 remove_watcher() i=6 2023-12-03T11:56:33.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.148+0000 7f37a1937800 20 remove_watcher() i=7 2023-12-03T11:56:33.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.156 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.161 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.204 INFO:tasks.workunit.client.0.smithi116.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:56:33.231 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:56:33.231 INFO:tasks.workunit.client.0.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:56:33.245 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.243+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 realm 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.245+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.246+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:33.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.246+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.246+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:56:33.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.246+0000 7f146ff1f800 20 searching for the correct realm 2023-12-03T11:56:33.257 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.255+0000 7f146bb4d640 0 -- 172.21.15.116:0/3374638896 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x5567b0f2f2f0 msgr2=0x5567b0f31700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:56:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:56:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:56:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:56:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.257+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 zone default found 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 4 Realm: () 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:56:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 zonegroup default 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 started sync module instance, tier type = 2023-12-03T11:56:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.258+0000 7f146ff1f800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.264+0000 7f146c34e640 0 -- 172.21.15.116:0/3374638896 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f14600264e0 msgr2=0x7f1460018400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=0 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=1 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=2 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=3 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=4 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=5 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.267+0000 7f146ff1f800 20 add_watcher() i=6 2023-12-03T11:56:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.268+0000 7f146ff1f800 20 add_watcher() i=7 2023-12-03T11:56:33.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.268+0000 7f146ff1f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:56:33.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.270+0000 7f1431fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:33.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.270+0000 7f146ff1f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:56:33.271 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.270+0000 7f146ff1f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:56:33.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f146ff1f800 5 note: GC not initialized 2023-12-03T11:56:33.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f13a8ff9640 20 reqs_thread_entry: start 2023-12-03T11:56:33.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f146ff1f800 20 init_complete bucket index max shards: 11 2023-12-03T11:56:33.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f146ff1f800 20 Filter name: none 2023-12-03T11:56:33.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f139e7fc640 20 reqs_thread_entry: start 2023-12-03T11:56:33.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.276+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:56:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.277+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.278+0000 7f146ff1f800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:56:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.278+0000 7f146ff1f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:56:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.279+0000 7f146ff1f800 20 rados->read ofs=0 len=0 2023-12-03T11:56:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.279+0000 7f146ff1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:56:33.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.280+0000 7f1431fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:56:33.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.283+0000 7f146ff1f800 20 remove_watcher() i=0 2023-12-03T11:56:33.284 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.283+0000 7f146ff1f800 2 removed watcher, disabling cache 2023-12-03T11:56:33.285 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.284+0000 7f146ff1f800 20 remove_watcher() i=1 2023-12-03T11:56:33.286 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.285+0000 7f146ff1f800 20 remove_watcher() i=2 2023-12-03T11:56:33.287 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.285+0000 7f146ff1f800 20 remove_watcher() i=3 2023-12-03T11:56:33.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.286+0000 7f146ff1f800 20 remove_watcher() i=4 2023-12-03T11:56:33.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.287+0000 7f146ff1f800 20 remove_watcher() i=5 2023-12-03T11:56:33.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.288+0000 7f146ff1f800 20 remove_watcher() i=6 2023-12-03T11:56:33.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-12-03T11:56:33.289+0000 7f146ff1f800 20 remove_watcher() i=7 2023-12-03T11:56:33.295 INFO:tasks.workunit.client.0.smithi116.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2023-12-03T11:56:33.295 INFO:tasks.workunit.client.0.smithi116.stderr:Deleting bucket a-bucket 2023-12-03T11:56:33.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.297 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.300 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.313 INFO:tasks.workunit.client.0.smithi116.stderr:Deleting bucket myver 2023-12-03T11:56:33.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.314 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2023-12-03T11:56:33.359 INFO:tasks.workunit.client.0.smithi116.stderr: warnings.warn( 2023-12-03T11:56:33.392 INFO:tasks.workunit.client.0.smithi116.stderr:Completed resharding tests 2023-12-03T11:56:33.392 INFO:tasks.workunit.client.0.smithi116.stdout:Could not connect to the endpoint URL: "http://localhost:80/" 2023-12-03T11:56:33.392 INFO:tasks.workunit.client.0.smithi116.stdout:Could not connect to the endpoint URL: "http://localhost:8000/" 2023-12-03T11:56:33.392 INFO:tasks.workunit.client.0.smithi116.stdout:connected to https://localhost:443 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ deactivate 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin ']' 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ export PATH 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset _OLD_VIRTUAL_PATH 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n '' ']' 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ hash -r 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -n '' ']' 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset VIRTUAL_ENV 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' '!' '' = nondestructive ']' 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ unset -f deactivate 2023-12-03T11:56:33.456 INFO:tasks.workunit.client.0.smithi116.stderr:+ echo OK. 2023-12-03T11:56:33.457 INFO:tasks.workunit.client.0.smithi116.stdout:OK. 2023-12-03T11:56:33.458 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:56:33.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:56:33.525 INFO:tasks.workunit:Stopping ['rgw/run-reshard.sh'] on client.0... 2023-12-03T11:56:33.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:56:33.888 DEBUG:teuthology.parallel:result is None 2023-12-03T11:56:33.889 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:56:33.915 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:56:33.915 DEBUG:teuthology.orchestra.run.smithi116:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:56:33.969 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:56:33.970 INFO:teuthology.run_tasks:Running task s3tests-java... 2023-12-03T11:56:33.984 DEBUG:tasks.s3tests_java:S3 Tests Java: __INIT__ 2023-12-03T11:56:33.984 DEBUG:tasks.s3tests_java:S3 Tests Java: SETUP 2023-12-03T11:56:33.984 INFO:tasks.s3tests_java:S3 Tests Java: Downloading test suite... 2023-12-03T11:56:33.985 DEBUG:teuthology.orchestra.run.smithi116:> git clone -b ceph-master https://github.com/ceph/java_s3tests.git /home/ubuntu/cephtest/s3-tests-java 2023-12-03T11:56:34.025 INFO:teuthology.orchestra.run.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-java'... 2023-12-03T11:56:38.811 INFO:tasks.s3tests_java:S3 Tests Java: Installing required packages... 2023-12-03T11:56:38.811 DEBUG:teuthology.orchestra.run.smithi116:> /home/ubuntu/cephtest/s3-tests-java/bootstrap.sh 2023-12-03T11:56:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:/home/ubuntu/cephtest/s3-tests-java/bootstrap.sh: missing required RPM packages. Installing via sudo. 2023-12-03T11:56:42.253 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:16:13 ago on Sun 03 Dec 2023 11:40:29 AM UTC. 2023-12-03T11:56:42.419 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================== 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================== 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 424 k 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk-devel x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 9.3 M 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-12-03T11:56:42.430 INFO:teuthology.orchestra.run.smithi116.stdout: ModemManager-glib x86_64 1.20.2-1.el9 CentOS-BaseOS 334 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: adwaita-cursor-theme noarch 40.1.1-3.el9 CentOS-AppStream 625 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: adwaita-icon-theme noarch 40.1.1-3.el9 CentOS-AppStream 11 M 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: alsa-lib x86_64 1.2.9-1.el9 CentOS-AppStream 499 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: at-spi2-atk x86_64 2.38.0-4.el9 CentOS-AppStream 86 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: at-spi2-core x86_64 2.40.3-1.el9 CentOS-AppStream 176 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: atk x86_64 2.36.0-5.el9 CentOS-AppStream 270 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: avahi-glib x86_64 0.8-12.el9 appstream 15 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: bluez-libs x86_64 5.64-1.el9 CentOS-BaseOS 83 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: bubblewrap x86_64 0.4.1-6.el9 CentOS-BaseOS 49 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: colord-libs x86_64 1.4.5-4.el9 CentOS-AppStream 228 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: copy-jdk-configs noarch 4.0-3.el9 CentOS-AppStream 27 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: cups-libs x86_64 1:2.3.3op2-22.el9 CentOS-BaseOS 262 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: exempi x86_64 2.6.0-0.2.20211007gite23c213.el9 CentOS-AppStream 524 k 2023-12-03T11:56:42.431 INFO:teuthology.orchestra.run.smithi116.stdout: exiv2-libs x86_64 0.27.5-2.el9 CentOS-AppStream 780 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: fdk-aac-free x86_64 2.0.0-8.el9 CentOS-AppStream 324 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: flac-libs x86_64 1.3.3-12.el9 appstream 218 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak-selinux noarch 1.12.8-1.el9 CentOS-AppStream 20 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak-session-helper x86_64 1.12.8-1.el9 CentOS-AppStream 73 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gdk-pixbuf2-modules x86_64 2.42.6-3.el9 CentOS-AppStream 84 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: geoclue2 x86_64 2.6.0-7.el9 CentOS-AppStream 122 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: giflib x86_64 5.2.1-9.el9 CentOS-AppStream 47 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: graphene x86_64 1.10.6-2.el9 CentOS-AppStream 63 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gsm x86_64 1.0.19-6.el9 CentOS-AppStream 33 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gstreamer1 x86_64 1.22.1-2.el9 CentOS-AppStream 1.4 M 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gstreamer1-plugins-base x86_64 1.22.1-1.el9 CentOS-AppStream 2.2 M 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gtk-update-icon-cache x86_64 3.24.31-2.el9 CentOS-AppStream 34 k 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: gtk3 x86_64 3.24.31-2.el9 CentOS-AppStream 4.8 M 2023-12-03T11:56:42.432 INFO:teuthology.orchestra.run.smithi116.stdout: hicolor-icon-theme noarch 0.17-13.el9 CentOS-AppStream 65 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: iso-codes noarch 4.6.0-3.el9 CentOS-AppStream 3.3 M 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk-headless x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 32 M 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: javapackages-filesystem noarch 6.0.0-4.el9 CentOS-AppStream 10 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: jbigkit-libs x86_64 2.1-23.el9 CentOS-AppStream 52 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: lcms2 x86_64 2.12-3.el9 CentOS-AppStream 166 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libX11-xcb x86_64 1.7.0-8.el9 appstream 11 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXcomposite x86_64 0.4.5-7.el9 CentOS-AppStream 23 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXcursor x86_64 1.2.0-7.el9 CentOS-AppStream 30 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXdamage x86_64 1.1.5-7.el9 CentOS-AppStream 22 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXfixes x86_64 5.0.3-16.el9 CentOS-AppStream 19 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXi x86_64 1.7.10-8.el9 CentOS-AppStream 38 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXinerama x86_64 1.1.4-10.el9 CentOS-AppStream 14 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXrandr x86_64 1.5.2-8.el9 CentOS-AppStream 27 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXtst x86_64 1.2.3-16.el9 CentOS-AppStream 20 k 2023-12-03T11:56:42.433 INFO:teuthology.orchestra.run.smithi116.stdout: libXv x86_64 1.0.11-16.el9 CentOS-AppStream 18 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libXxf86vm x86_64 1.1.4-18.el9 CentOS-AppStream 18 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libasyncns x86_64 0.8-22.el9 CentOS-AppStream 29 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra x86_64 0.30-27.el9 CentOS-AppStream 85 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra-gtk3 x86_64 0.30-27.el9 CentOS-AppStream 31 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libdrm x86_64 2.4.117-1.el9 CentOS-AppStream 158 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libepoxy x86_64 1.5.5-4.el9 CentOS-AppStream 240 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libexif x86_64 0.6.22-6.el9 CentOS-AppStream 431 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libfontenc x86_64 1.1.3-17.el9 CentOS-AppStream 30 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libgexiv2 x86_64 0.12.3-1.el9 CentOS-AppStream 82 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd x86_64 1:1.3.4-1.el9 CentOS-AppStream 134 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd-egl x86_64 1:1.3.4-1.el9 CentOS-AppStream 36 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd-glx x86_64 1:1.3.4-1.el9 CentOS-AppStream 141 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libgsf x86_64 1.14.47-5.el9 CentOS-AppStream 245 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libgxps x86_64 0.3.2-3.el9 CentOS-AppStream 78 k 2023-12-03T11:56:42.434 INFO:teuthology.orchestra.run.smithi116.stdout: libiptcdata x86_64 1.0.5-9.el9 CentOS-AppStream 61 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libldac x86_64 2.0.2.3-10.el9 CentOS-AppStream 40 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libnotify x86_64 0.7.9-8.el9 CentOS-AppStream 43 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libogg x86_64 2:1.3.4-6.el9 CentOS-AppStream 32 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libosinfo x86_64 1.10.0-1.el9 CentOS-AppStream 312 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libpciaccess x86_64 0.16-6.el9 CentOS-BaseOS 27 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libsbc x86_64 1.4-9.el9 CentOS-AppStream 44 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libsndfile x86_64 1.0.31-8.el9 CentOS-AppStream 205 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libtheora x86_64 1:1.1.1-31.el9 CentOS-AppStream 163 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libtiff x86_64 4.4.0-10.el9 CentOS-AppStream 196 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-ltdl x86_64 2.4.6-45.el9 CentOS-AppStream 36 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libtracker-sparql x86_64 3.1.2-3.el9 CentOS-AppStream 316 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libvisual x86_64 1:0.4.0-34.el9 CentOS-AppStream 143 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libvorbis x86_64 1:1.3.7-5.el9 CentOS-AppStream 192 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-client x86_64 1.21.0-1.el9 CentOS-AppStream 32 k 2023-12-03T11:56:42.435 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-cursor x86_64 1.21.0-1.el9 CentOS-AppStream 18 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-egl x86_64 1.21.0-1.el9 CentOS-AppStream 12 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-server x86_64 1.21.0-1.el9 CentOS-AppStream 41 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: libwebp x86_64 1.2.0-8.el9 CentOS-AppStream 276 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: libxkbcommon x86_64 1.0.3-4.el9 CentOS-AppStream 132 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: libxshmfence x86_64 1.3-10.el9 CentOS-AppStream 12 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: low-memory-monitor x86_64 2.1-4.el9 CentOS-AppStream 35 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: lua-posix x86_64 35.0-8.el9 CentOS-AppStream 131 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libEGL x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 126 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libGL x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 167 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libgbm x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 36 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libglapi x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 44 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-vulkan-drivers x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 9.7 M 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: mkfontscale x86_64 1.2.1-3.el9 CentOS-AppStream 31 k 2023-12-03T11:56:42.436 INFO:teuthology.orchestra.run.smithi116.stdout: openjpeg2 x86_64 2.4.0-7.el9 CentOS-AppStream 162 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: opus x86_64 1.3.1-10.el9 CentOS-AppStream 199 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: orc x86_64 0.4.31-6.el9 CentOS-AppStream 183 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: osinfo-db noarch 20230518-1.el9 CentOS-AppStream 273 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: osinfo-db-tools x86_64 1.10.0-1.el9 CentOS-AppStream 68 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: ostree-libs x86_64 2023.7-2.el9 CentOS-AppStream 477 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-libs x86_64 0.3.67-2.el9 CentOS-AppStream 1.7 M 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: poppler x86_64 21.01.0-19.el9 CentOS-AppStream 1.1 M 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: poppler-data noarch 0.4.9-9.el9 CentOS-AppStream 1.8 M 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: poppler-glib x86_64 21.01.0-19.el9 CentOS-AppStream 151 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: pulseaudio-libs x86_64 15.0-2.el9 CentOS-AppStream 666 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: pulseaudio-utils x86_64 15.0-2.el9 CentOS-AppStream 72 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: rtkit x86_64 0.11-28.el9 CentOS-AppStream 55 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: sound-theme-freedesktop noarch 0.8-17.el9 CentOS-AppStream 377 k 2023-12-03T11:56:42.437 INFO:teuthology.orchestra.run.smithi116.stdout: totem-pl-parser x86_64 3.26.6-2.el9 CentOS-AppStream 130 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: ttmkfdir x86_64 3.0.9-65.el9 CentOS-AppStream 52 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: tzdata-java noarch 2023c-1.el9 CentOS-AppStream 149 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: upower x86_64 0.99.11-11.el9 CentOS-AppStream 168 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: vulkan-loader x86_64 1.3.250.1-1.el9 CentOS-AppStream 140 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: webrtc-audio-processing x86_64 0.3.1-8.el9 CentOS-AppStream 305 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: wireplumber x86_64 0.4.14-1.el9 CentOS-AppStream 83 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: wireplumber-libs x86_64 0.4.14-1.el9 CentOS-AppStream 337 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-dbus-proxy x86_64 0.1.3-1.el9 CentOS-AppStream 41 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-desktop-portal x86_64 1.12.4-1.el9 CentOS-AppStream 365 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: xkeyboard-config noarch 2.33-2.el9 CentOS-AppStream 780 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: xorg-x11-fonts-Type1 noarch 7.5-33.el9 CentOS-AppStream 499 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: dconf x86_64 0.40.0-6.el9 CentOS-AppStream 108 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: exiv2 x86_64 0.27.5-2.el9 CentOS-AppStream 978 k 2023-12-03T11:56:42.438 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak x86_64 1.12.8-1.el9 CentOS-AppStream 1.7 M 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: gtk2 x86_64 2.24.33-8.el9 CentOS-AppStream 3.5 M 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra-gtk2 x86_64 0.30-27.el9 CentOS-AppStream 25 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: p11-kit-server x86_64 0.24.1-2.el9 CentOS-AppStream 198 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire x86_64 0.3.67-2.el9 CentOS-AppStream 86 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-alsa x86_64 0.3.67-2.el9 CentOS-AppStream 55 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-jack-audio-connection-kit x86_64 0.3.67-2.el9 CentOS-AppStream 126 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-pulseaudio x86_64 0.3.67-2.el9 CentOS-AppStream 18 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: tracker x86_64 3.1.2-3.el9 CentOS-AppStream 537 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: tracker-miners x86_64 3.1.2-3.el9 CentOS-AppStream 887 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-desktop-portal-gtk x86_64 1.12.0-3.el9 CentOS-AppStream 130 k 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================== 2023-12-03T11:56:42.439 INFO:teuthology.orchestra.run.smithi116.stdout:Install 129 Packages 2023-12-03T11:56:42.440 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:56:42.440 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 102 M 2023-12-03T11:56:42.440 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 381 M 2023-12-03T11:56:42.440 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-12-03T11:56:42.854 INFO:teuthology.orchestra.run.smithi116.stdout:(1/129): alsa-lib-1.2.9-1.el9.x86_64.rpm 1.7 MB/s | 499 kB 00:00 2023-12-03T11:56:42.879 INFO:teuthology.orchestra.run.smithi116.stdout:(2/129): adwaita-cursor-theme-40.1.1-3.el9.noar 2.0 MB/s | 625 kB 00:00 2023-12-03T11:56:42.904 INFO:teuthology.orchestra.run.smithi116.stdout:(3/129): at-spi2-atk-2.38.0-4.el9.x86_64.rpm 1.7 MB/s | 86 kB 00:00 2023-12-03T11:56:42.930 INFO:teuthology.orchestra.run.smithi116.stdout:(4/129): at-spi2-core-2.40.3-1.el9.x86_64.rpm 3.5 MB/s | 176 kB 00:00 2023-12-03T11:56:42.955 INFO:teuthology.orchestra.run.smithi116.stdout:(5/129): atk-2.36.0-5.el9.x86_64.rpm 5.4 MB/s | 270 kB 00:00 2023-12-03T11:56:42.989 INFO:teuthology.orchestra.run.smithi116.stdout:(6/129): colord-libs-1.4.5-4.el9.x86_64.rpm 4.0 MB/s | 228 kB 00:00 2023-12-03T11:56:43.014 INFO:teuthology.orchestra.run.smithi116.stdout:(7/129): copy-jdk-configs-4.0-3.el9.noarch.rpm 481 kB/s | 27 kB 00:00 2023-12-03T11:56:43.041 INFO:teuthology.orchestra.run.smithi116.stdout:(8/129): dconf-0.40.0-6.el9.x86_64.rpm 2.2 MB/s | 108 kB 00:00 2023-12-03T11:56:43.148 INFO:teuthology.orchestra.run.smithi116.stdout:(9/129): adwaita-icon-theme-40.1.1-3.el9.noarch 18 MB/s | 11 MB 00:00 2023-12-03T11:56:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:(10/129): exempi-2.6.0-0.2.20211007gite23c213.e 3.3 MB/s | 524 kB 00:00 2023-12-03T11:56:43.207 INFO:teuthology.orchestra.run.smithi116.stdout:(11/129): exiv2-0.27.5-2.el9.x86_64.rpm 5.7 MB/s | 978 kB 00:00 2023-12-03T11:56:43.241 INFO:teuthology.orchestra.run.smithi116.stdout:(12/129): exiv2-libs-0.27.5-2.el9.x86_64.rpm 8.4 MB/s | 780 kB 00:00 2023-12-03T11:56:43.266 INFO:teuthology.orchestra.run.smithi116.stdout:(13/129): fdk-aac-free-2.0.0-8.el9.x86_64.rpm 5.5 MB/s | 324 kB 00:00 2023-12-03T11:56:43.308 INFO:teuthology.orchestra.run.smithi116.stdout:(14/129): flatpak-1.12.8-1.el9.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-12-03T11:56:43.325 INFO:teuthology.orchestra.run.smithi116.stdout:(15/129): flatpak-selinux-1.12.8-1.el9.noarch.r 246 kB/s | 20 kB 00:00 2023-12-03T11:56:43.350 INFO:teuthology.orchestra.run.smithi116.stdout:(16/129): flatpak-session-helper-1.12.8-1.el9.x 881 kB/s | 73 kB 00:00 2023-12-03T11:56:43.375 INFO:teuthology.orchestra.run.smithi116.stdout:(17/129): gdk-pixbuf2-modules-2.42.6-3.el9.x86_ 1.6 MB/s | 84 kB 00:00 2023-12-03T11:56:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:(18/129): geoclue2-2.6.0-7.el9.x86_64.rpm 1.6 MB/s | 122 kB 00:00 2023-12-03T11:56:43.417 INFO:teuthology.orchestra.run.smithi116.stdout:(19/129): giflib-5.2.1-9.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2023-12-03T11:56:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:(20/129): graphene-1.10.6-2.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-03T11:56:43.451 INFO:teuthology.orchestra.run.smithi116.stdout:(21/129): gsm-1.0.19-6.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2023-12-03T11:56:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:(22/129): gstreamer1-1.22.1-2.el9.x86_64.rpm 19 MB/s | 1.4 MB 00:00 2023-12-03T11:56:43.518 INFO:teuthology.orchestra.run.smithi116.stdout:(23/129): gtk-update-icon-cache-3.24.31-2.el9.x 526 kB/s | 34 kB 00:00 2023-12-03T11:56:43.577 INFO:teuthology.orchestra.run.smithi116.stdout:(24/129): gstreamer1-plugins-base-1.22.1-1.el9. 18 MB/s | 2.2 MB 00:00 2023-12-03T11:56:43.635 INFO:teuthology.orchestra.run.smithi116.stdout:(25/129): gtk2-2.24.33-8.el9.x86_64.rpm 24 MB/s | 3.5 MB 00:00 2023-12-03T11:56:43.752 INFO:teuthology.orchestra.run.smithi116.stdout:(26/129): gtk3-3.24.31-2.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-12-03T11:56:43.778 INFO:teuthology.orchestra.run.smithi116.stdout:(27/129): hicolor-icon-theme-0.17-13.el9.noarch 331 kB/s | 65 kB 00:00 2023-12-03T11:56:43.845 INFO:teuthology.orchestra.run.smithi116.stdout:(28/129): iso-codes-4.6.0-3.el9.noarch.rpm 16 MB/s | 3.3 MB 00:00 2023-12-03T11:56:43.870 INFO:teuthology.orchestra.run.smithi116.stdout:(29/129): java-1.8.0-openjdk-1.8.0.362.b09-4.el 3.6 MB/s | 424 kB 00:00 2023-12-03T11:56:44.054 INFO:teuthology.orchestra.run.smithi116.stdout:(30/129): java-1.8.0-openjdk-devel-1.8.0.362.b0 34 MB/s | 9.3 MB 00:00 2023-12-03T11:56:44.079 INFO:teuthology.orchestra.run.smithi116.stdout:(31/129): javapackages-filesystem-6.0.0-4.el9.n 51 kB/s | 10 kB 00:00 2023-12-03T11:56:44.279 INFO:teuthology.orchestra.run.smithi116.stdout:(32/129): jbigkit-libs-2.1-23.el9.x86_64.rpm 262 kB/s | 52 kB 00:00 2023-12-03T11:56:44.304 INFO:teuthology.orchestra.run.smithi116.stdout:(33/129): lcms2-2.12-3.el9.x86_64.rpm 747 kB/s | 166 kB 00:00 2023-12-03T11:56:44.713 INFO:teuthology.orchestra.run.smithi116.stdout:(34/129): java-1.8.0-openjdk-headless-1.8.0.362 38 MB/s | 32 MB 00:00 2023-12-03T11:56:44.730 INFO:teuthology.orchestra.run.smithi116.stdout:(35/129): libXcomposite-0.4.5-7.el9.x86_64.rpm 51 kB/s | 23 kB 00:00 2023-12-03T11:56:44.747 INFO:teuthology.orchestra.run.smithi116.stdout:(36/129): libXcursor-1.2.0-7.el9.x86_64.rpm 67 kB/s | 30 kB 00:00 2023-12-03T11:56:44.764 INFO:teuthology.orchestra.run.smithi116.stdout:(37/129): libXdamage-1.1.5-7.el9.x86_64.rpm 433 kB/s | 22 kB 00:00 2023-12-03T11:56:44.781 INFO:teuthology.orchestra.run.smithi116.stdout:(38/129): libXi-1.7.10-8.el9.x86_64.rpm 1.1 MB/s | 38 kB 00:00 2023-12-03T11:56:44.798 INFO:teuthology.orchestra.run.smithi116.stdout:(39/129): libXfixes-5.0.3-16.el9.x86_64.rpm 369 kB/s | 19 kB 00:00 2023-12-03T11:56:44.814 INFO:teuthology.orchestra.run.smithi116.stdout:(40/129): libXinerama-1.1.4-10.el9.x86_64.rpm 279 kB/s | 14 kB 00:00 2023-12-03T11:56:44.832 INFO:teuthology.orchestra.run.smithi116.stdout:(41/129): libXrandr-1.5.2-8.el9.x86_64.rpm 545 kB/s | 27 kB 00:00 2023-12-03T11:56:44.848 INFO:teuthology.orchestra.run.smithi116.stdout:(42/129): libXv-1.0.11-16.el9.x86_64.rpm 542 kB/s | 18 kB 00:00 2023-12-03T11:56:44.874 INFO:teuthology.orchestra.run.smithi116.stdout:(43/129): libXtst-1.2.3-16.el9.x86_64.rpm 347 kB/s | 20 kB 00:00 2023-12-03T11:56:44.899 INFO:teuthology.orchestra.run.smithi116.stdout:(44/129): libXxf86vm-1.1.4-18.el9.x86_64.rpm 263 kB/s | 18 kB 00:00 2023-12-03T11:56:44.924 INFO:teuthology.orchestra.run.smithi116.stdout:(45/129): libasyncns-0.8-22.el9.x86_64.rpm 390 kB/s | 29 kB 00:00 2023-12-03T11:56:44.950 INFO:teuthology.orchestra.run.smithi116.stdout:(46/129): libcanberra-0.30-27.el9.x86_64.rpm 1.7 MB/s | 85 kB 00:00 2023-12-03T11:56:44.975 INFO:teuthology.orchestra.run.smithi116.stdout:(47/129): libcanberra-gtk2-0.30-27.el9.x86_64.r 330 kB/s | 25 kB 00:00 2023-12-03T11:56:45.000 INFO:teuthology.orchestra.run.smithi116.stdout:(48/129): libcanberra-gtk3-0.30-27.el9.x86_64.r 408 kB/s | 31 kB 00:00 2023-12-03T11:56:45.025 INFO:teuthology.orchestra.run.smithi116.stdout:(49/129): libdrm-2.4.117-1.el9.x86_64.rpm 3.1 MB/s | 158 kB 00:00 2023-12-03T11:56:45.050 INFO:teuthology.orchestra.run.smithi116.stdout:(50/129): libepoxy-1.5.5-4.el9.x86_64.rpm 3.1 MB/s | 240 kB 00:00 2023-12-03T11:56:45.084 INFO:teuthology.orchestra.run.smithi116.stdout:(51/129): libexif-0.6.22-6.el9.x86_64.rpm 5.1 MB/s | 431 kB 00:00 2023-12-03T11:56:45.109 INFO:teuthology.orchestra.run.smithi116.stdout:(52/129): libfontenc-1.1.3-17.el9.x86_64.rpm 518 kB/s | 30 kB 00:00 2023-12-03T11:56:45.134 INFO:teuthology.orchestra.run.smithi116.stdout:(53/129): libgexiv2-0.12.3-1.el9.x86_64.rpm 985 kB/s | 82 kB 00:00 2023-12-03T11:56:45.160 INFO:teuthology.orchestra.run.smithi116.stdout:(54/129): libglvnd-1.3.4-1.el9.x86_64.rpm 1.7 MB/s | 134 kB 00:00 2023-12-03T11:56:45.194 INFO:teuthology.orchestra.run.smithi116.stdout:(55/129): libglvnd-glx-1.3.4-1.el9.x86_64.rpm 2.4 MB/s | 141 kB 00:00 2023-12-03T11:56:45.218 INFO:teuthology.orchestra.run.smithi116.stdout:(56/129): libglvnd-egl-1.3.4-1.el9.x86_64.rpm 427 kB/s | 36 kB 00:00 2023-12-03T11:56:45.252 INFO:teuthology.orchestra.run.smithi116.stdout:(57/129): libgsf-1.14.47-5.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2023-12-03T11:56:45.278 INFO:teuthology.orchestra.run.smithi116.stdout:(58/129): libgxps-0.3.2-3.el9.x86_64.rpm 937 kB/s | 78 kB 00:00 2023-12-03T11:56:45.303 INFO:teuthology.orchestra.run.smithi116.stdout:(59/129): libiptcdata-1.0.5-9.el9.x86_64.rpm 1.2 MB/s | 61 kB 00:00 2023-12-03T11:56:45.328 INFO:teuthology.orchestra.run.smithi116.stdout:(60/129): libldac-2.0.2.3-10.el9.x86_64.rpm 535 kB/s | 40 kB 00:00 2023-12-03T11:56:45.353 INFO:teuthology.orchestra.run.smithi116.stdout:(61/129): libnotify-0.7.9-8.el9.x86_64.rpm 567 kB/s | 43 kB 00:00 2023-12-03T11:56:45.379 INFO:teuthology.orchestra.run.smithi116.stdout:(62/129): libogg-1.3.4-6.el9.x86_64.rpm 653 kB/s | 32 kB 00:00 2023-12-03T11:56:45.404 INFO:teuthology.orchestra.run.smithi116.stdout:(63/129): libosinfo-1.10.0-1.el9.x86_64.rpm 4.1 MB/s | 312 kB 00:00 2023-12-03T11:56:45.429 INFO:teuthology.orchestra.run.smithi116.stdout:(64/129): libsbc-1.4-9.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2023-12-03T11:56:45.454 INFO:teuthology.orchestra.run.smithi116.stdout:(65/129): libsndfile-1.0.31-8.el9.x86_64.rpm 4.0 MB/s | 205 kB 00:00 2023-12-03T11:56:45.479 INFO:teuthology.orchestra.run.smithi116.stdout:(66/129): libtheora-1.1.1-31.el9.x86_64.rpm 2.1 MB/s | 163 kB 00:00 2023-12-03T11:56:45.505 INFO:teuthology.orchestra.run.smithi116.stdout:(67/129): libtiff-4.4.0-10.el9.x86_64.rpm 2.6 MB/s | 196 kB 00:00 2023-12-03T11:56:45.530 INFO:teuthology.orchestra.run.smithi116.stdout:(68/129): libtool-ltdl-2.4.6-45.el9.x86_64.rpm 725 kB/s | 36 kB 00:00 2023-12-03T11:56:45.555 INFO:teuthology.orchestra.run.smithi116.stdout:(69/129): libtracker-sparql-3.1.2-3.el9.x86_64. 4.1 MB/s | 316 kB 00:00 2023-12-03T11:56:45.580 INFO:teuthology.orchestra.run.smithi116.stdout:(70/129): libvisual-0.4.0-34.el9.x86_64.rpm 1.9 MB/s | 143 kB 00:00 2023-12-03T11:56:45.614 INFO:teuthology.orchestra.run.smithi116.stdout:(71/129): libwayland-client-1.21.0-1.el9.x86_64 554 kB/s | 32 kB 00:00 2023-12-03T11:56:45.639 INFO:teuthology.orchestra.run.smithi116.stdout:(72/129): libvorbis-1.3.7-5.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2023-12-03T11:56:45.656 INFO:teuthology.orchestra.run.smithi116.stdout:(73/129): libwayland-cursor-1.21.0-1.el9.x86_64 242 kB/s | 18 kB 00:00 2023-12-03T11:56:45.682 INFO:teuthology.orchestra.run.smithi116.stdout:(74/129): libwayland-egl-1.21.0-1.el9.x86_64.rp 177 kB/s | 12 kB 00:00 2023-12-03T11:56:45.710 INFO:teuthology.orchestra.run.smithi116.stdout:(75/129): libwayland-server-1.21.0-1.el9.x86_64 812 kB/s | 41 kB 00:00 2023-12-03T11:56:45.732 INFO:teuthology.orchestra.run.smithi116.stdout:(76/129): libwebp-1.2.0-8.el9.x86_64.rpm 3.6 MB/s | 276 kB 00:00 2023-12-03T11:56:45.757 INFO:teuthology.orchestra.run.smithi116.stdout:(77/129): libxkbcommon-1.0.3-4.el9.x86_64.rpm 1.7 MB/s | 132 kB 00:00 2023-12-03T11:56:45.774 INFO:teuthology.orchestra.run.smithi116.stdout:(78/129): libxshmfence-1.3-10.el9.x86_64.rpm 279 kB/s | 12 kB 00:00 2023-12-03T11:56:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:(79/129): low-memory-monitor-2.1-4.el9.x86_64.r 599 kB/s | 35 kB 00:00 2023-12-03T11:56:45.817 INFO:teuthology.orchestra.run.smithi116.stdout:(80/129): lua-posix-35.0-8.el9.x86_64.rpm 2.2 MB/s | 131 kB 00:00 2023-12-03T11:56:45.842 INFO:teuthology.orchestra.run.smithi116.stdout:(81/129): mesa-libEGL-23.3.0~rc2-1.el9.x86_64.r 2.5 MB/s | 126 kB 00:00 2023-12-03T11:56:45.867 INFO:teuthology.orchestra.run.smithi116.stdout:(82/129): mesa-libGL-23.3.0~rc2-1.el9.x86_64.rp 2.2 MB/s | 167 kB 00:00 2023-12-03T11:56:45.892 INFO:teuthology.orchestra.run.smithi116.stdout:(83/129): mesa-libgbm-23.3.0~rc2-1.el9.x86_64.r 487 kB/s | 36 kB 00:00 2023-12-03T11:56:45.909 INFO:teuthology.orchestra.run.smithi116.stdout:(84/129): mesa-libglapi-23.3.0~rc2-1.el9.x86_64 1.0 MB/s | 44 kB 00:00 2023-12-03T11:56:46.085 INFO:teuthology.orchestra.run.smithi116.stdout:(85/129): mesa-vulkan-drivers-23.3.0~rc2-1.el9. 45 MB/s | 9.7 MB 00:00 2023-12-03T11:56:46.109 INFO:teuthology.orchestra.run.smithi116.stdout:(86/129): mkfontscale-1.2.1-3.el9.x86_64.rpm 146 kB/s | 31 kB 00:00 2023-12-03T11:56:46.135 INFO:teuthology.orchestra.run.smithi116.stdout:(87/129): openjpeg2-2.4.0-7.el9.x86_64.rpm 727 kB/s | 162 kB 00:00 2023-12-03T11:56:46.160 INFO:teuthology.orchestra.run.smithi116.stdout:(88/129): opus-1.3.1-10.el9.x86_64.rpm 2.6 MB/s | 199 kB 00:00 2023-12-03T11:56:46.194 INFO:teuthology.orchestra.run.smithi116.stdout:(89/129): orc-0.4.31-6.el9.x86_64.rpm 3.1 MB/s | 183 kB 00:00 2023-12-03T11:56:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:(90/129): osinfo-db-20230518-1.el9.noarch.rpm 3.2 MB/s | 273 kB 00:00 2023-12-03T11:56:46.244 INFO:teuthology.orchestra.run.smithi116.stdout:(91/129): osinfo-db-tools-1.10.0-1.el9.x86_64.r 824 kB/s | 68 kB 00:00 2023-12-03T11:56:46.269 INFO:teuthology.orchestra.run.smithi116.stdout:(92/129): p11-kit-server-0.24.1-2.el9.x86_64.rp 3.9 MB/s | 198 kB 00:00 2023-12-03T11:56:46.303 INFO:teuthology.orchestra.run.smithi116.stdout:(93/129): ostree-libs-2023.7-2.el9.x86_64.rpm 5.6 MB/s | 477 kB 00:00 2023-12-03T11:56:46.328 INFO:teuthology.orchestra.run.smithi116.stdout:(94/129): pipewire-0.3.67-2.el9.x86_64.rpm 1.0 MB/s | 86 kB 00:00 2023-12-03T11:56:46.354 INFO:teuthology.orchestra.run.smithi116.stdout:(95/129): pipewire-alsa-0.3.67-2.el9.x86_64.rpm 658 kB/s | 55 kB 00:00 2023-12-03T11:56:46.379 INFO:teuthology.orchestra.run.smithi116.stdout:(96/129): pipewire-jack-audio-connection-kit-0. 2.5 MB/s | 126 kB 00:00 2023-12-03T11:56:46.421 INFO:teuthology.orchestra.run.smithi116.stdout:(97/129): pipewire-libs-0.3.67-2.el9.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2023-12-03T11:56:46.446 INFO:teuthology.orchestra.run.smithi116.stdout:(98/129): pipewire-pulseaudio-0.3.67-2.el9.x86_ 198 kB/s | 18 kB 00:00 2023-12-03T11:56:46.488 INFO:teuthology.orchestra.run.smithi116.stdout:(99/129): poppler-21.01.0-19.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2023-12-03T11:56:46.513 INFO:teuthology.orchestra.run.smithi116.stdout:(100/129): poppler-glib-21.01.0-19.el9.x86_64.r 2.3 MB/s | 151 kB 00:00 2023-12-03T11:56:46.555 INFO:teuthology.orchestra.run.smithi116.stdout:(101/129): poppler-data-0.4.9-9.el9.noarch.rpm 17 MB/s | 1.8 MB 00:00 2023-12-03T11:56:46.580 INFO:teuthology.orchestra.run.smithi116.stdout:(102/129): pulseaudio-libs-15.0-2.el9.x86_64.rp 7.1 MB/s | 666 kB 00:00 2023-12-03T11:56:46.606 INFO:teuthology.orchestra.run.smithi116.stdout:(103/129): pulseaudio-utils-15.0-2.el9.x86_64.r 800 kB/s | 72 kB 00:00 2023-12-03T11:56:46.623 INFO:teuthology.orchestra.run.smithi116.stdout:(104/129): rtkit-0.11-28.el9.x86_64.rpm 1.3 MB/s | 55 kB 00:00 2023-12-03T11:56:46.648 INFO:teuthology.orchestra.run.smithi116.stdout:(105/129): sound-theme-freedesktop-0.8-17.el9.n 5.5 MB/s | 377 kB 00:00 2023-12-03T11:56:46.673 INFO:teuthology.orchestra.run.smithi116.stdout:(106/129): totem-pl-parser-3.26.6-2.el9.x86_64. 1.9 MB/s | 130 kB 00:00 2023-12-03T11:56:46.907 INFO:teuthology.orchestra.run.smithi116.stdout:(107/129): tracker-3.1.2-3.el9.x86_64.rpm 2.0 MB/s | 537 kB 00:00 2023-12-03T11:56:46.941 INFO:teuthology.orchestra.run.smithi116.stdout:(108/129): tracker-miners-3.1.2-3.el9.x86_64.rp 3.0 MB/s | 887 kB 00:00 2023-12-03T11:56:46.966 INFO:teuthology.orchestra.run.smithi116.stdout:(109/129): ttmkfdir-3.0.9-65.el9.x86_64.rpm 179 kB/s | 52 kB 00:00 2023-12-03T11:56:46.991 INFO:teuthology.orchestra.run.smithi116.stdout:(110/129): tzdata-java-2023c-1.el9.noarch.rpm 1.8 MB/s | 149 kB 00:00 2023-12-03T11:56:47.058 INFO:teuthology.orchestra.run.smithi116.stdout:(111/129): upower-0.99.11-11.el9.x86_64.rpm 1.8 MB/s | 168 kB 00:00 2023-12-03T11:56:47.092 INFO:teuthology.orchestra.run.smithi116.stdout:(112/129): vulkan-loader-1.3.250.1-1.el9.x86_64 1.1 MB/s | 140 kB 00:00 2023-12-03T11:56:47.159 INFO:teuthology.orchestra.run.smithi116.stdout:(113/129): webrtc-audio-processing-0.3.1-8.el9. 1.8 MB/s | 305 kB 00:00 2023-12-03T11:56:47.184 INFO:teuthology.orchestra.run.smithi116.stdout:(114/129): wireplumber-0.4.14-1.el9.x86_64.rpm 663 kB/s | 83 kB 00:00 2023-12-03T11:56:47.251 INFO:teuthology.orchestra.run.smithi116.stdout:(115/129): wireplumber-libs-0.4.14-1.el9.x86_64 2.1 MB/s | 337 kB 00:00 2023-12-03T11:56:47.276 INFO:teuthology.orchestra.run.smithi116.stdout:(116/129): xdg-dbus-proxy-0.1.3-1.el9.x86_64.rp 354 kB/s | 41 kB 00:00 2023-12-03T11:56:47.318 INFO:teuthology.orchestra.run.smithi116.stdout:(117/129): xdg-desktop-portal-1.12.4-1.el9.x86_ 2.7 MB/s | 365 kB 00:00 2023-12-03T11:56:47.343 INFO:teuthology.orchestra.run.smithi116.stdout:(118/129): xdg-desktop-portal-gtk-1.12.0-3.el9. 1.4 MB/s | 130 kB 00:00 2023-12-03T11:56:47.460 INFO:teuthology.orchestra.run.smithi116.stdout:(119/129): xkeyboard-config-2.33-2.el9.noarch.r 4.2 MB/s | 780 kB 00:00 2023-12-03T11:56:47.511 INFO:teuthology.orchestra.run.smithi116.stdout:(120/129): xorg-x11-fonts-Type1-7.5-33.el9.noar 2.5 MB/s | 499 kB 00:00 2023-12-03T11:56:47.544 INFO:teuthology.orchestra.run.smithi116.stdout:(121/129): ModemManager-glib-1.20.2-1.el9.x86_6 1.6 MB/s | 334 kB 00:00 2023-12-03T11:56:47.570 INFO:teuthology.orchestra.run.smithi116.stdout:(122/129): bluez-libs-5.64-1.el9.x86_64.rpm 763 kB/s | 83 kB 00:00 2023-12-03T11:56:47.586 INFO:teuthology.orchestra.run.smithi116.stdout:(123/129): bubblewrap-0.4.1-6.el9.x86_64.rpm 656 kB/s | 49 kB 00:00 2023-12-03T11:56:47.612 INFO:teuthology.orchestra.run.smithi116.stdout:(124/129): cups-libs-2.3.3op2-22.el9.x86_64.rpm 3.8 MB/s | 262 kB 00:00 2023-12-03T11:56:47.628 INFO:teuthology.orchestra.run.smithi116.stdout:(125/129): libpciaccess-0.16-6.el9.x86_64.rpm 643 kB/s | 27 kB 00:00 2023-12-03T11:56:47.654 INFO:teuthology.orchestra.run.smithi116.stdout:(126/129): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.3 MB/s | 90 kB 00:00 2023-12-03T11:56:47.704 INFO:teuthology.orchestra.run.smithi116.stdout:(127/129): libX11-xcb-1.7.0-8.el9.x86_64.rpm 219 kB/s | 11 kB 00:00 2023-12-03T11:56:47.721 INFO:teuthology.orchestra.run.smithi116.stdout:(128/129): avahi-glib-0.8-12.el9.x86_64.rpm 136 kB/s | 15 kB 00:00 2023-12-03T11:56:47.788 INFO:teuthology.orchestra.run.smithi116.stdout:(129/129): flac-libs-1.3.3-12.el9.x86_64.rpm 1.6 MB/s | 218 kB 00:00 2023-12-03T11:56:47.791 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:56:47.791 INFO:teuthology.orchestra.run.smithi116.stdout:Total 19 MB/s | 102 MB 00:05 2023-12-03T11:56:47.824 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-12-03T11:56:47.911 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-12-03T11:56:47.911 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-12-03T11:56:48.918 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-12-03T11:56:48.918 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-12-03T11:56:53.082 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: copy-jdk-configs-4.0-3.el9.noarch 1/1 2023-12-03T11:56:53.138 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el9.x8 1/1 2023-12-03T11:56:53.287 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-12-03T11:56:53.357 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXi-1.7.10-8.el9.x86_64 1/129 2023-12-03T11:56:53.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libwayland-client-1.21.0-1.el9.x86_64 2/129 2023-12-03T11:56:53.430 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libogg-2:1.3.4-6.el9.x86_64 3/129 2023-12-03T11:56:53.460 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libvorbis-1:1.3.7-5.el9.x86_64 4/129 2023-12-03T11:56:53.589 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXfixes-5.0.3-16.el9.x86_64 5/129 2023-12-03T11:56:53.662 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : atk-2.36.0-5.el9.x86_64 6/129 2023-12-03T11:56:53.689 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : alsa-lib-1.2.9-1.el9.x86_64 7/129 2023-12-03T11:56:53.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libX11-xcb-1.7.0-8.el9.x86_64 8/129 2023-12-03T11:56:53.750 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cups-libs-1:2.3.3op2-22.el9.x86_64 9/129 2023-12-03T11:56:53.782 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : opus-1.3.1-10.el9.x86_64 10/129 2023-12-03T11:56:53.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libxshmfence-1.3-10.el9.x86_64 11/129 2023-12-03T11:56:53.835 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-ltdl-2.4.6-45.el9.x86_64 12/129 2023-12-03T11:56:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXcomposite-0.4.5-7.el9.x86_64 13/129 2023-12-03T11:56:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lcms2-2.12-3.el9.x86_64 14/129 2023-12-03T11:56:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gstreamer1-1.22.1-2.el9.x86_64 15/129 2023-12-03T11:56:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXcursor-1.2.0-7.el9.x86_64 16/129 2023-12-03T11:56:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXdamage-1.1.5-7.el9.x86_64 17/129 2023-12-03T11:56:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libwayland-cursor-1.21.0-1.el9.x86_64 18/129 2023-12-03T11:56:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXtst-1.2.3-16.el9.x86_64 19/129 2023-12-03T11:56:54.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mesa-libglapi-23.3.0~rc2-1.el9.x86_64 20/129 2023-12-03T11:56:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libwayland-server-1.21.0-1.el9.x86_64 21/129 2023-12-03T11:56:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libwayland-egl-1.21.0-1.el9.x86_64 22/129 2023-12-03T11:56:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libglvnd-1:1.3.4-1.el9.x86_64 23/129 2023-12-03T11:56:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXrandr-1.5.2-8.el9.x86_64 24/129 2023-12-03T11:56:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXinerama-1.1.4-10.el9.x86_64 25/129 2023-12-03T11:56:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : hicolor-icon-theme-0.17-13.el9.noarch 26/129 2023-12-03T11:56:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gtk-update-icon-cache-3.24.31-2.el9.x86_64 27/129 2023-12-03T11:56:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : dconf-0.40.0-6.el9.x86_64 28/129 2023-12-03T11:56:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 28/129 2023-12-03T11:56:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : at-spi2-core-2.40.3-1.el9.x86_64 29/129 2023-12-03T11:56:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : at-spi2-atk-2.38.0-4.el9.x86_64 30/129 2023-12-03T11:56:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : colord-libs-1.4.5-4.el9.x86_64 31/129 2023-12-03T11:56:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtheora-1:1.1.1-31.el9.x86_64 32/129 2023-12-03T11:56:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flac-libs-1.3.3-12.el9.x86_64 33/129 2023-12-03T11:56:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : exiv2-libs-0.27.5-2.el9.x86_64 34/129 2023-12-03T11:56:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : exiv2-0.27.5-2.el9.x86_64 35/129 2023-12-03T11:56:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgexiv2-0.12.3-1.el9.x86_64 36/129 2023-12-03T11:56:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : avahi-glib-0.8-12.el9.x86_64 37/129 2023-12-03T11:56:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 38/129 2023-12-03T11:56:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpciaccess-0.16-6.el9.x86_64 39/129 2023-12-03T11:56:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libdrm-2.4.117-1.el9.x86_64 40/129 2023-12-03T11:56:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mesa-libgbm-23.3.0~rc2-1.el9.x86_64 41/129 2023-12-03T11:56:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libglvnd-egl-1:1.3.4-1.el9.x86_64 42/129 2023-12-03T11:56:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mesa-libEGL-23.3.0~rc2-1.el9.x86_64 43/129 2023-12-03T11:56:55.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : vulkan-loader-1.3.250.1-1.el9.x86_64 44/129 2023-12-03T11:56:55.410 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 45/129 2023-12-03T11:56:55.435 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bubblewrap-0.4.1-6.el9.x86_64 46/129 2023-12-03T11:56:55.467 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bluez-libs-5.64-1.el9.x86_64 47/129 2023-12-03T11:56:55.661 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ModemManager-glib-1.20.2-1.el9.x86_64 48/129 2023-12-03T11:56:55.698 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xkeyboard-config-2.33-2.el9.noarch 49/129 2023-12-03T11:56:55.739 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libxkbcommon-1.0.3-4.el9.x86_64 50/129 2023-12-03T11:56:55.768 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xdg-dbus-proxy-0.1.3-1.el9.x86_64 51/129 2023-12-03T11:56:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : webrtc-audio-processing-0.3.1-8.el9.x86_64 52/129 2023-12-03T11:56:55.847 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : upower-0.99.11-11.el9.x86_64 53/129 2023-12-03T11:56:55.860 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: upower-0.99.11-11.el9.x86_64 53/129 2023-12-03T11:56:55.860 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/graphical.target.wants/upower.service → /usr/lib/systemd/system/upower.service. 2023-12-03T11:56:55.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:56:56.043 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : tzdata-java-2023c-1.el9.noarch 54/129 2023-12-03T11:56:56.110 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ttmkfdir-3.0.9-65.el9.x86_64 55/129 2023-12-03T11:56:56.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : totem-pl-parser-3.26.6-2.el9.x86_64 56/129 2023-12-03T11:56:56.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : sound-theme-freedesktop-0.8-17.el9.noarch 57/129 2023-12-03T11:56:56.202 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: sound-theme-freedesktop-0.8-17.el9.noarch 57/129 2023-12-03T11:56:56.618 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:56:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:56:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:56:56.654 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/graphical.target.wants/rtkit-daemon.service → /usr/lib/systemd/system/rtkit-daemon.service. 2023-12-03T11:56:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:56:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : poppler-data-0.4.9-9.el9.noarch 59/129 2023-12-03T11:56:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : p11-kit-server-0.24.1-2.el9.x86_64 60/129 2023-12-03T11:56:56.924 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ostree-libs-2023.7-2.el9.x86_64 61/129 2023-12-03T11:56:57.393 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : osinfo-db-tools-1.10.0-1.el9.x86_64 62/129 2023-12-03T11:56:57.452 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : osinfo-db-20230518-1.el9.noarch 63/129 2023-12-03T11:56:57.491 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libosinfo-1.10.0-1.el9.x86_64 64/129 2023-12-03T11:56:57.524 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : orc-0.4.31-6.el9.x86_64 65/129 2023-12-03T11:56:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openjpeg2-2.4.0-7.el9.x86_64 66/129 2023-12-03T11:56:57.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-posix-35.0-8.el9.x86_64 67/129 2023-12-03T11:56:57.634 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : copy-jdk-configs-4.0-3.el9.noarch 68/129 2023-12-03T11:56:57.655 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : low-memory-monitor-2.1-4.el9.x86_64 69/129 2023-12-03T11:56:57.667 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: low-memory-monitor-2.1-4.el9.x86_64 69/129 2023-12-03T11:56:57.667 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/basic.target.wants/low-memory-monitor.service → /usr/lib/systemd/system/low-memory-monitor.service. 2023-12-03T11:56:57.667 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:56:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libwebp-1.2.0-8.el9.x86_64 70/129 2023-12-03T11:56:57.742 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libvisual-1:0.4.0-34.el9.x86_64 71/129 2023-12-03T11:56:57.772 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libsbc-1.4-9.el9.x86_64 72/129 2023-12-03T11:56:57.802 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libnotify-0.7.9-8.el9.x86_64 73/129 2023-12-03T11:56:58.201 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:56:58.231 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:56:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:56:58.280 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libldac-2.0.2.3-10.el9.x86_64 75/129 2023-12-03T11:56:58.338 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libiptcdata-1.0.5-9.el9.x86_64 76/129 2023-12-03T11:56:58.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgsf-1.14.47-5.el9.x86_64 77/129 2023-12-03T11:56:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libfontenc-1.1.3-17.el9.x86_64 78/129 2023-12-03T11:56:58.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mkfontscale-1.2.1-3.el9.x86_64 79/129 2023-12-03T11:56:58.477 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xorg-x11-fonts-Type1-7.5-33.el9.noarch 80/129 2023-12-03T11:56:58.758 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xorg-x11-fonts-Type1-7.5-33.el9.noarch 80/129 2023-12-03T11:56:58.804 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libexif-0.6.22-6.el9.x86_64 81/129 2023-12-03T11:56:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libepoxy-1.5.5-4.el9.x86_64 82/129 2023-12-03T11:56:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libasyncns-0.8-22.el9.x86_64 83/129 2023-12-03T11:56:58.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXxf86vm-1.1.4-18.el9.x86_64 84/129 2023-12-03T11:56:58.918 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libglvnd-glx-1:1.3.4-1.el9.x86_64 85/129 2023-12-03T11:56:58.943 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : mesa-libGL-23.3.0~rc2-1.el9.x86_64 86/129 2023-12-03T11:56:58.970 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libXv-1.0.11-16.el9.x86_64 87/129 2023-12-03T11:56:58.998 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jbigkit-libs-2.1-23.el9.x86_64 88/129 2023-12-03T11:56:59.036 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtiff-4.4.0-10.el9.x86_64 89/129 2023-12-03T11:56:59.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 90/129 2023-12-03T11:56:59.115 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgxps-0.3.2-3.el9.x86_64 91/129 2023-12-03T11:56:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : poppler-21.01.0-19.el9.x86_64 92/129 2023-12-03T11:56:59.182 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : poppler-glib-21.01.0-19.el9.x86_64 93/129 2023-12-03T11:56:59.771 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : javapackages-filesystem-6.0.0-4.el9.noarch 94/129 2023-12-03T11:56:59.809 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 95/129 2023-12-03T11:57:00.458 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 95/129 2023-12-03T11:57:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : iso-codes-4.6.0-3.el9.noarch 96/129 2023-12-03T11:57:00.530 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gsm-1.0.19-6.el9.x86_64 97/129 2023-12-03T11:57:00.597 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libsndfile-1.0.31-8.el9.x86_64 98/129 2023-12-03T11:57:00.637 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pulseaudio-libs-15.0-2.el9.x86_64 99/129 2023-12-03T11:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcanberra-0.30-27.el9.x86_64 100/129 2023-12-03T11:57:00.686 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcanberra-0.30-27.el9.x86_64 100/129 2023-12-03T11:57:00.725 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pulseaudio-utils-15.0-2.el9.x86_64 101/129 2023-12-03T11:57:00.832 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : graphene-1.10.6-2.el9.x86_64 102/129 2023-12-03T11:57:00.871 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gstreamer1-plugins-base-1.22.1-1.el9.x86_64 103/129 2023-12-03T11:57:00.898 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : giflib-5.2.1-9.el9.x86_64 104/129 2023-12-03T11:57:00.929 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flatpak-session-helper-1.12.8-1.el9.x86_64 105/129 2023-12-03T11:57:00.952 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flatpak-selinux-1.12.8-1.el9.noarch 106/129 2023-12-03T11:57:11.352 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: flatpak-selinux-1.12.8-1.el9.noarch 106/129 2023-12-03T11:57:11.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fdk-aac-free-2.0.0-8.el9.x86_64 107/129 2023-12-03T11:57:11.854 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:57:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:57:11.896 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:57:11.896 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire.socket → /usr/lib/systemd/user/pipewire.socket. 2023-12-03T11:57:11.896 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:12.104 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pipewire-libs-0.3.67-2.el9.x86_64 109/129 2023-12-03T11:57:12.165 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : wireplumber-0.4.14-1.el9.x86_64 110/129 2023-12-03T11:57:12.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : wireplumber-libs-0.4.14-1.el9.x86_64 111/129 2023-12-03T11:57:12.720 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: flatpak-1.12.8-1.el9.x86_64 112/129 2023-12-03T11:57:12.802 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flatpak-1.12.8-1.el9.x86_64 112/129 2023-12-03T11:57:12.842 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xdg-desktop-portal-1.12.4-1.el9.x86_64 113/129 2023-12-03T11:57:12.874 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xdg-desktop-portal-1.12.4-1.el9.x86_64 113/129 2023-12-03T11:57:12.922 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 114/129 2023-12-03T11:57:12.991 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtracker-sparql-3.1.2-3.el9.x86_64 115/129 2023-12-03T11:57:13.021 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : tracker-3.1.2-3.el9.x86_64 116/129 2023-12-03T11:57:13.126 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: tracker-3.1.2-3.el9.x86_64 116/129 2023-12-03T11:57:13.168 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : tracker-miners-3.1.2-3.el9.x86_64 117/129 2023-12-03T11:57:13.257 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: tracker-miners-3.1.2-3.el9.x86_64 117/129 2023-12-03T11:57:16.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : adwaita-cursor-theme-40.1.1-3.el9.noarch 118/129 2023-12-03T11:57:16.153 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : adwaita-icon-theme-40.1.1-3.el9.noarch 119/129 2023-12-03T11:57:16.197 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcanberra-gtk3-0.30-27.el9.x86_64 120/129 2023-12-03T11:57:16.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 121/129 2023-12-03T11:57:16.402 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 121/129 2023-12-03T11:57:16.550 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gtk3-3.24.31-2.el9.x86_64 122/129 2023-12-03T11:57:16.587 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gtk2-2.24.33-8.el9.x86_64 123/129 2023-12-03T11:57:16.622 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcanberra-gtk2-0.30-27.el9.x86_64 124/129 2023-12-03T11:57:16.653 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 125/129 2023-12-03T11:57:16.892 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 125/129 2023-12-03T11:57:16.957 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 126/129 2023-12-03T11:57:16.968 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 126/129 2023-12-03T11:57:17.014 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pipewire-alsa-0.3.67-2.el9.x86_64 127/129 2023-12-03T11:57:17.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pipewire-jack-audio-connection-kit-0.3.67-2.el9. 128/129 2023-12-03T11:57:17.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:57:17.084 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:57:17.084 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire-pulse.socket → /usr/lib/systemd/user/pipewire-pulse.socket. 2023-12-03T11:57:17.084 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:17.106 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 129/129 2023-12-03T11:57:17.112 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: copy-jdk-configs-4.0-3.el9.noarch 129/129 2023-12-03T11:57:17.132 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 129/129 2023-12-03T11:57:17.142 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: wireplumber-0.4.14-1.el9.x86_64 129/129 2023-12-03T11:57:17.142 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/user/pipewire-session-manager.service → /usr/lib/systemd/user/wireplumber.service. 2023-12-03T11:57:17.142 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/user/pipewire.service.wants/wireplumber.service → /usr/lib/systemd/user/wireplumber.service. 2023-12-03T11:57:17.142 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:17.206 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 129/129 2023-12-03T11:57:17.277 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 129/129 2023-12-03T11:57:21.144 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:57:21.144 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : adwaita-cursor-theme-40.1.1-3.el9.noarch 1/129 2023-12-03T11:57:21.144 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : adwaita-icon-theme-40.1.1-3.el9.noarch 2/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : alsa-lib-1.2.9-1.el9.x86_64 3/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : at-spi2-atk-2.38.0-4.el9.x86_64 4/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : at-spi2-core-2.40.3-1.el9.x86_64 5/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : atk-2.36.0-5.el9.x86_64 6/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : colord-libs-1.4.5-4.el9.x86_64 7/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : copy-jdk-configs-4.0-3.el9.noarch 8/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : dconf-0.40.0-6.el9.x86_64 9/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 10/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : exiv2-0.27.5-2.el9.x86_64 11/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : exiv2-libs-0.27.5-2.el9.x86_64 12/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fdk-aac-free-2.0.0-8.el9.x86_64 13/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flatpak-1.12.8-1.el9.x86_64 14/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flatpak-selinux-1.12.8-1.el9.noarch 15/129 2023-12-03T11:57:21.145 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flatpak-session-helper-1.12.8-1.el9.x86_64 16/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 17/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : geoclue2-2.6.0-7.el9.x86_64 18/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : giflib-5.2.1-9.el9.x86_64 19/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : graphene-1.10.6-2.el9.x86_64 20/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gsm-1.0.19-6.el9.x86_64 21/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gstreamer1-1.22.1-2.el9.x86_64 22/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gstreamer1-plugins-base-1.22.1-1.el9.x86_64 23/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gtk-update-icon-cache-3.24.31-2.el9.x86_64 24/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gtk2-2.24.33-8.el9.x86_64 25/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gtk3-3.24.31-2.el9.x86_64 26/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : hicolor-icon-theme-0.17-13.el9.noarch 27/129 2023-12-03T11:57:21.146 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iso-codes-4.6.0-3.el9.noarch 28/129 2023-12-03T11:57:21.147 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 29/129 2023-12-03T11:57:21.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 30/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 31/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : javapackages-filesystem-6.0.0-4.el9.noarch 32/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jbigkit-libs-2.1-23.el9.x86_64 33/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lcms2-2.12-3.el9.x86_64 34/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXcomposite-0.4.5-7.el9.x86_64 35/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXcursor-1.2.0-7.el9.x86_64 36/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXdamage-1.1.5-7.el9.x86_64 37/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXfixes-5.0.3-16.el9.x86_64 38/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXi-1.7.10-8.el9.x86_64 39/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXinerama-1.1.4-10.el9.x86_64 40/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXrandr-1.5.2-8.el9.x86_64 41/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXtst-1.2.3-16.el9.x86_64 42/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXv-1.0.11-16.el9.x86_64 43/129 2023-12-03T11:57:21.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libXxf86vm-1.1.4-18.el9.x86_64 44/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libasyncns-0.8-22.el9.x86_64 45/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcanberra-0.30-27.el9.x86_64 46/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcanberra-gtk2-0.30-27.el9.x86_64 47/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcanberra-gtk3-0.30-27.el9.x86_64 48/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libdrm-2.4.117-1.el9.x86_64 49/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libepoxy-1.5.5-4.el9.x86_64 50/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libexif-0.6.22-6.el9.x86_64 51/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfontenc-1.1.3-17.el9.x86_64 52/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgexiv2-0.12.3-1.el9.x86_64 53/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libglvnd-1:1.3.4-1.el9.x86_64 54/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libglvnd-egl-1:1.3.4-1.el9.x86_64 55/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libglvnd-glx-1:1.3.4-1.el9.x86_64 56/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgsf-1.14.47-5.el9.x86_64 57/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgxps-0.3.2-3.el9.x86_64 58/129 2023-12-03T11:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libiptcdata-1.0.5-9.el9.x86_64 59/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libldac-2.0.2.3-10.el9.x86_64 60/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libnotify-0.7.9-8.el9.x86_64 61/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libogg-2:1.3.4-6.el9.x86_64 62/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libosinfo-1.10.0-1.el9.x86_64 63/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsbc-1.4-9.el9.x86_64 64/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsndfile-1.0.31-8.el9.x86_64 65/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtheora-1:1.1.1-31.el9.x86_64 66/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtiff-4.4.0-10.el9.x86_64 67/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-ltdl-2.4.6-45.el9.x86_64 68/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtracker-sparql-3.1.2-3.el9.x86_64 69/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libvisual-1:0.4.0-34.el9.x86_64 70/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libvorbis-1:1.3.7-5.el9.x86_64 71/129 2023-12-03T11:57:21.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libwayland-client-1.21.0-1.el9.x86_64 72/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libwayland-cursor-1.21.0-1.el9.x86_64 73/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libwayland-egl-1.21.0-1.el9.x86_64 74/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libwayland-server-1.21.0-1.el9.x86_64 75/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libwebp-1.2.0-8.el9.x86_64 76/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libxkbcommon-1.0.3-4.el9.x86_64 77/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libxshmfence-1.3-10.el9.x86_64 78/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : low-memory-monitor-2.1-4.el9.x86_64 79/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-posix-35.0-8.el9.x86_64 80/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mesa-libEGL-23.3.0~rc2-1.el9.x86_64 81/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mesa-libGL-23.3.0~rc2-1.el9.x86_64 82/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mesa-libgbm-23.3.0~rc2-1.el9.x86_64 83/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mesa-libglapi-23.3.0~rc2-1.el9.x86_64 84/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 85/129 2023-12-03T11:57:21.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : mkfontscale-1.2.1-3.el9.x86_64 86/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openjpeg2-2.4.0-7.el9.x86_64 87/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : opus-1.3.1-10.el9.x86_64 88/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : orc-0.4.31-6.el9.x86_64 89/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : osinfo-db-20230518-1.el9.noarch 90/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : osinfo-db-tools-1.10.0-1.el9.x86_64 91/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ostree-libs-2023.7-2.el9.x86_64 92/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : p11-kit-server-0.24.1-2.el9.x86_64 93/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pipewire-0.3.67-2.el9.x86_64 94/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pipewire-alsa-0.3.67-2.el9.x86_64 95/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pipewire-jack-audio-connection-kit-0.3.67-2.el9. 96/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pipewire-libs-0.3.67-2.el9.x86_64 97/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pipewire-pulseaudio-0.3.67-2.el9.x86_64 98/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : poppler-21.01.0-19.el9.x86_64 99/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : poppler-data-0.4.9-9.el9.noarch 100/129 2023-12-03T11:57:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : poppler-glib-21.01.0-19.el9.x86_64 101/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pulseaudio-libs-15.0-2.el9.x86_64 102/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pulseaudio-utils-15.0-2.el9.x86_64 103/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rtkit-0.11-28.el9.x86_64 104/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : sound-theme-freedesktop-0.8-17.el9.noarch 105/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : totem-pl-parser-3.26.6-2.el9.x86_64 106/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : tracker-3.1.2-3.el9.x86_64 107/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : tracker-miners-3.1.2-3.el9.x86_64 108/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ttmkfdir-3.0.9-65.el9.x86_64 109/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : tzdata-java-2023c-1.el9.noarch 110/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : upower-0.99.11-11.el9.x86_64 111/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : vulkan-loader-1.3.250.1-1.el9.x86_64 112/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : webrtc-audio-processing-0.3.1-8.el9.x86_64 113/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : wireplumber-0.4.14-1.el9.x86_64 114/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : wireplumber-libs-0.4.14-1.el9.x86_64 115/129 2023-12-03T11:57:21.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xdg-dbus-proxy-0.1.3-1.el9.x86_64 116/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xdg-desktop-portal-1.12.4-1.el9.x86_64 117/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 118/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xkeyboard-config-2.33-2.el9.noarch 119/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xorg-x11-fonts-Type1-7.5-33.el9.noarch 120/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ModemManager-glib-1.20.2-1.el9.x86_64 121/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bluez-libs-5.64-1.el9.x86_64 122/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bubblewrap-0.4.1-6.el9.x86_64 123/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cups-libs-1:2.3.3op2-22.el9.x86_64 124/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpciaccess-0.16-6.el9.x86_64 125/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 126/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : avahi-glib-0.8-12.el9.x86_64 127/129 2023-12-03T11:57:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flac-libs-1.3.3-12.el9.x86_64 128/129 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libX11-xcb-1.7.0-8.el9.x86_64 129/129 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: ModemManager-glib-1.20.2-1.el9.x86_64 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: adwaita-cursor-theme-40.1.1-3.el9.noarch 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: adwaita-icon-theme-40.1.1-3.el9.noarch 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: alsa-lib-1.2.9-1.el9.x86_64 2023-12-03T11:57:23.300 INFO:teuthology.orchestra.run.smithi116.stdout: at-spi2-atk-2.38.0-4.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: at-spi2-core-2.40.3-1.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: atk-2.36.0-5.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: avahi-glib-0.8-12.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: bluez-libs-5.64-1.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: bubblewrap-0.4.1-6.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: colord-libs-1.4.5-4.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: copy-jdk-configs-4.0-3.el9.noarch 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: cups-libs-1:2.3.3op2-22.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: dconf-0.40.0-6.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: exiv2-0.27.5-2.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: exiv2-libs-0.27.5-2.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: fdk-aac-free-2.0.0-8.el9.x86_64 2023-12-03T11:57:23.301 INFO:teuthology.orchestra.run.smithi116.stdout: flac-libs-1.3.3-12.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak-1.12.8-1.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak-selinux-1.12.8-1.el9.noarch 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: flatpak-session-helper-1.12.8-1.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: geoclue2-2.6.0-7.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: giflib-5.2.1-9.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: graphene-1.10.6-2.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gsm-1.0.19-6.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gstreamer1-1.22.1-2.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gstreamer1-plugins-base-1.22.1-1.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gtk-update-icon-cache-3.24.31-2.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gtk2-2.24.33-8.el9.x86_64 2023-12-03T11:57:23.302 INFO:teuthology.orchestra.run.smithi116.stdout: gtk3-3.24.31-2.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: hicolor-icon-theme-0.17-13.el9.noarch 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: iso-codes-4.6.0-3.el9.noarch 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: javapackages-filesystem-6.0.0-4.el9.noarch 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: jbigkit-libs-2.1-23.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: lcms2-2.12-3.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libX11-xcb-1.7.0-8.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libXcomposite-0.4.5-7.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libXcursor-1.2.0-7.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libXdamage-1.1.5-7.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libXfixes-5.0.3-16.el9.x86_64 2023-12-03T11:57:23.303 INFO:teuthology.orchestra.run.smithi116.stdout: libXi-1.7.10-8.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libXinerama-1.1.4-10.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libXrandr-1.5.2-8.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libXtst-1.2.3-16.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libXv-1.0.11-16.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libXxf86vm-1.1.4-18.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libasyncns-0.8-22.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra-0.30-27.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra-gtk2-0.30-27.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libcanberra-gtk3-0.30-27.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libdrm-2.4.117-1.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libepoxy-1.5.5-4.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libexif-0.6.22-6.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libfontenc-1.1.3-17.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libgexiv2-0.12.3-1.el9.x86_64 2023-12-03T11:57:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd-1:1.3.4-1.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd-egl-1:1.3.4-1.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libglvnd-glx-1:1.3.4-1.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libgsf-1.14.47-5.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libgxps-0.3.2-3.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libiptcdata-1.0.5-9.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libldac-2.0.2.3-10.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libnotify-0.7.9-8.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libogg-2:1.3.4-6.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libosinfo-1.10.0-1.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libpciaccess-0.16-6.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libsbc-1.4-9.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libsndfile-1.0.31-8.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libtheora-1:1.1.1-31.el9.x86_64 2023-12-03T11:57:23.305 INFO:teuthology.orchestra.run.smithi116.stdout: libtiff-4.4.0-10.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-ltdl-2.4.6-45.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libtracker-sparql-3.1.2-3.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libvisual-1:0.4.0-34.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libvorbis-1:1.3.7-5.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-client-1.21.0-1.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-cursor-1.21.0-1.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-egl-1.21.0-1.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libwayland-server-1.21.0-1.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libwebp-1.2.0-8.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libxkbcommon-1.0.3-4.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: libxshmfence-1.3-10.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: low-memory-monitor-2.1-4.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: lua-posix-35.0-8.el9.x86_64 2023-12-03T11:57:23.306 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libEGL-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libGL-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libgbm-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-libglapi-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: mkfontscale-1.2.1-3.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: openjpeg2-2.4.0-7.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: opus-1.3.1-10.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: orc-0.4.31-6.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: osinfo-db-20230518-1.el9.noarch 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: osinfo-db-tools-1.10.0-1.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: ostree-libs-2023.7-2.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: p11-kit-server-0.24.1-2.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-0.3.67-2.el9.x86_64 2023-12-03T11:57:23.307 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-alsa-0.3.67-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-jack-audio-connection-kit-0.3.67-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-libs-0.3.67-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: pipewire-pulseaudio-0.3.67-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: poppler-21.01.0-19.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: poppler-data-0.4.9-9.el9.noarch 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: poppler-glib-21.01.0-19.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: pulseaudio-libs-15.0-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: pulseaudio-utils-15.0-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: rtkit-0.11-28.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: sound-theme-freedesktop-0.8-17.el9.noarch 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: totem-pl-parser-3.26.6-2.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: tracker-3.1.2-3.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: tracker-miners-3.1.2-3.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: ttmkfdir-3.0.9-65.el9.x86_64 2023-12-03T11:57:23.308 INFO:teuthology.orchestra.run.smithi116.stdout: tzdata-java-2023c-1.el9.noarch 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: upower-0.99.11-11.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: vulkan-loader-1.3.250.1-1.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: webrtc-audio-processing-0.3.1-8.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: wireplumber-0.4.14-1.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: wireplumber-libs-0.4.14-1.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-dbus-proxy-0.1.3-1.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-desktop-portal-1.12.4-1.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: xkeyboard-config-2.33-2.el9.noarch 2023-12-03T11:57:23.309 INFO:teuthology.orchestra.run.smithi116.stdout: xorg-x11-fonts-Type1-7.5-33.el9.noarch 2023-12-03T11:57:23.310 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:23.310 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-12-03T11:57:23.654 INFO:teuthology.orchestra.run.smithi116.stdout: Gradle will be installed under /opt/gradle 2023-12-03T11:57:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:--2023-12-03 11:57:23-- https://services.gradle.org/distributions/gradle-6.0.1-bin.zip 2023-12-03T11:57:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:Resolving services.gradle.org (services.gradle.org)... 104.17.129.37, 104.17.128.37, 2606:4700::6811:8025, ... 2023-12-03T11:57:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:Connecting to services.gradle.org (services.gradle.org)|104.17.129.37|:443... connected. 2023-12-03T11:57:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:HTTP request sent, awaiting response... 301 Moved Permanently 2023-12-03T11:57:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:Location: https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip [following] 2023-12-03T11:57:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:--2023-12-03 11:57:23-- https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip 2023-12-03T11:57:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:Resolving github.com (github.com)... 140.82.112.4 2023-12-03T11:57:23.959 INFO:teuthology.orchestra.run.smithi116.stderr:Connecting to github.com (github.com)|140.82.112.4|:443... connected. 2023-12-03T11:57:24.123 INFO:teuthology.orchestra.run.smithi116.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-03T11:57:24.124 INFO:teuthology.orchestra.run.smithi116.stderr:Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20231203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231203T115724Z&X-Amz-Expires=300&X-Amz-Signature=93124ac9ee1a0deea3c369c4ce738b5f8ff8b7ec53c529d52283d6d52f9d6ce7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream [following] 2023-12-03T11:57:24.124 INFO:teuthology.orchestra.run.smithi116.stderr:--2023-12-03 11:57:24-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20231203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231203T115724Z&X-Amz-Expires=300&X-Amz-Signature=93124ac9ee1a0deea3c369c4ce738b5f8ff8b7ec53c529d52283d6d52f9d6ce7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream 2023-12-03T11:57:24.124 INFO:teuthology.orchestra.run.smithi116.stderr:Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... 2023-12-03T11:57:24.139 INFO:teuthology.orchestra.run.smithi116.stderr:Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected. 2023-12-03T11:57:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-03T11:57:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:Length: 95129672 (91M) [application/octet-stream] 2023-12-03T11:57:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:Saving to: ‘gradle-6.0.1-bin.zip’ 2023-12-03T11:57:24.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T11:57:24.231 INFO:teuthology.orchestra.run.smithi116.stderr: 0K .......... .......... .......... .......... .......... 0% 1.67M 54s 2023-12-03T11:57:24.235 INFO:teuthology.orchestra.run.smithi116.stderr: 50K .......... .......... .......... .......... .......... 0% 11.4M 31s 2023-12-03T11:57:24.246 INFO:teuthology.orchestra.run.smithi116.stderr: 100K .......... .......... .......... .......... .......... 0% 4.19M 28s 2023-12-03T11:57:24.249 INFO:teuthology.orchestra.run.smithi116.stderr: 150K .......... .......... .......... .......... .......... 0% 16.0M 22s 2023-12-03T11:57:24.252 INFO:teuthology.orchestra.run.smithi116.stderr: 200K .......... .......... .......... .......... .......... 0% 18.7M 19s 2023-12-03T11:57:24.254 INFO:teuthology.orchestra.run.smithi116.stderr: 250K .......... .......... .......... .......... .......... 0% 23.7M 16s 2023-12-03T11:57:24.262 INFO:teuthology.orchestra.run.smithi116.stderr: 300K .......... .......... .......... .......... .......... 0% 6.55M 16s 2023-12-03T11:57:24.264 INFO:teuthology.orchestra.run.smithi116.stderr: 350K .......... .......... .......... .......... .......... 0% 24.4M 14s 2023-12-03T11:57:24.265 INFO:teuthology.orchestra.run.smithi116.stderr: 400K .......... .......... .......... .......... .......... 0% 32.6M 13s 2023-12-03T11:57:24.266 INFO:teuthology.orchestra.run.smithi116.stderr: 450K .......... .......... .......... .......... .......... 0% 37.4M 12s 2023-12-03T11:57:24.268 INFO:teuthology.orchestra.run.smithi116.stderr: 500K .......... .......... .......... .......... .......... 0% 46.2M 11s 2023-12-03T11:57:24.269 INFO:teuthology.orchestra.run.smithi116.stderr: 550K .......... .......... .......... .......... .......... 0% 31.7M 10s 2023-12-03T11:57:24.275 INFO:teuthology.orchestra.run.smithi116.stderr: 600K .......... .......... .......... .......... .......... 0% 8.05M 10s 2023-12-03T11:57:24.276 INFO:teuthology.orchestra.run.smithi116.stderr: 650K .......... .......... .......... .......... .......... 0% 39.0M 10s 2023-12-03T11:57:24.278 INFO:teuthology.orchestra.run.smithi116.stderr: 700K .......... .......... .......... .......... .......... 0% 40.9M 9s 2023-12-03T11:57:24.281 INFO:teuthology.orchestra.run.smithi116.stderr: 750K .......... .......... .......... .......... .......... 0% 45.3M 9s 2023-12-03T11:57:24.281 INFO:teuthology.orchestra.run.smithi116.stderr: 800K .......... .......... .......... .......... .......... 0% 54.2M 8s 2023-12-03T11:57:24.281 INFO:teuthology.orchestra.run.smithi116.stderr: 850K .......... .......... .......... .......... .......... 0% 90.6M 8s 2023-12-03T11:57:24.281 INFO:teuthology.orchestra.run.smithi116.stderr: 900K .......... .......... .......... .......... .......... 1% 116M 8s 2023-12-03T11:57:24.283 INFO:teuthology.orchestra.run.smithi116.stderr: 950K .......... .......... .......... .......... .......... 1% 54.1M 7s 2023-12-03T11:57:24.283 INFO:teuthology.orchestra.run.smithi116.stderr: 1000K .......... .......... .......... .......... .......... 1% 46.0M 7s 2023-12-03T11:57:24.286 INFO:teuthology.orchestra.run.smithi116.stderr: 1050K .......... .......... .......... .......... .......... 1% 49.3M 7s 2023-12-03T11:57:24.286 INFO:teuthology.orchestra.run.smithi116.stderr: 1100K .......... .......... .......... .......... .......... 1% 111M 7s 2023-12-03T11:57:24.286 INFO:teuthology.orchestra.run.smithi116.stderr: 1150K .......... .......... .......... .......... .......... 1% 65.6M 6s 2023-12-03T11:57:24.286 INFO:teuthology.orchestra.run.smithi116.stderr: 1200K .......... .......... .......... .......... .......... 1% 98.3M 6s 2023-12-03T11:57:24.290 INFO:teuthology.orchestra.run.smithi116.stderr: 1250K .......... .......... .......... .......... .......... 1% 9.93M 6s 2023-12-03T11:57:24.292 INFO:teuthology.orchestra.run.smithi116.stderr: 1300K .......... .......... .......... .......... .......... 1% 41.1M 6s 2023-12-03T11:57:24.292 INFO:teuthology.orchestra.run.smithi116.stderr: 1350K .......... .......... .......... .......... .......... 1% 119M 6s 2023-12-03T11:57:25.595 INFO:teuthology.orchestra.run.smithi116.stderr: 1400K .......... .......... .......... .......... .......... 1% 50.2M 6s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1450K .......... .......... .......... .......... .......... 1% 107M 6s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1500K .......... .......... .......... .......... .......... 1% 110M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1550K .......... .......... .......... .......... .......... 1% 108M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1600K .......... .......... .......... .......... .......... 1% 105M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1650K .......... .......... .......... .......... .......... 1% 72.5M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1700K .......... .......... .......... .......... .......... 1% 121M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1750K .......... .......... .......... .......... .......... 1% 111M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1800K .......... .......... .......... .......... .......... 1% 125M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1850K .......... .......... .......... .......... .......... 2% 93.6M 5s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1900K .......... .......... .......... .......... .......... 2% 113M 4s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 1950K .......... .......... .......... .......... .......... 2% 76.7M 4s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 2000K .......... .......... .......... .......... .......... 2% 103M 4s 2023-12-03T11:57:25.596 INFO:teuthology.orchestra.run.smithi116.stderr: 2050K .......... .......... .......... .......... .......... 2% 44.5M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2100K .......... .......... .......... .......... .......... 2% 93.3M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2150K .......... .......... .......... .......... .......... 2% 79.8M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2200K .......... .......... .......... .......... .......... 2% 116M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2250K .......... .......... .......... .......... .......... 2% 115M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2300K .......... .......... .......... .......... .......... 2% 58.1M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2350K .......... .......... .......... .......... .......... 2% 83.3M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2400K .......... .......... .......... .......... .......... 2% 121M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2450K .......... .......... .......... .......... .......... 2% 87.6M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2500K .......... .......... .......... .......... .......... 2% 29.9M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2550K .......... .......... .......... .......... .......... 2% 132M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2600K .......... .......... .......... .......... .......... 2% 127M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2650K .......... .......... .......... .......... .......... 2% 110M 4s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2700K .......... .......... .......... .......... .......... 2% 116M 3s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2750K .......... .......... .......... .......... .......... 3% 121M 3s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2800K .......... .......... .......... .......... .......... 3% 109M 3s 2023-12-03T11:57:25.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2850K .......... .......... .......... .......... .......... 3% 88.1M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 2900K .......... .......... .......... .......... .......... 3% 125M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 2950K .......... .......... .......... .......... .......... 3% 116M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3000K .......... .......... .......... .......... .......... 3% 116M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3050K .......... .......... .......... .......... .......... 3% 107M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3100K .......... .......... .......... .......... .......... 3% 95.7M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3150K .......... .......... .......... .......... .......... 3% 138M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3200K .......... .......... .......... .......... .......... 3% 109M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3250K .......... .......... .......... .......... .......... 3% 102M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3300K .......... .......... .......... .......... .......... 3% 62.9M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3350K .......... .......... .......... .......... .......... 3% 99.7M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3400K .......... .......... .......... .......... .......... 3% 126M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3450K .......... .......... .......... .......... .......... 3% 91.2M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3500K .......... .......... .......... .......... .......... 3% 128M 3s 2023-12-03T11:57:25.598 INFO:teuthology.orchestra.run.smithi116.stderr: 3550K .......... .......... .......... .......... .......... 3% 125M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3600K .......... .......... .......... .......... .......... 3% 91.8M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3650K .......... .......... .......... .......... .......... 3% 119M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3700K .......... .......... .......... .......... .......... 4% 114M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3750K .......... .......... .......... .......... .......... 4% 111M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3800K .......... .......... .......... .......... .......... 4% 120M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3850K .......... .......... .......... .......... .......... 4% 109M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3900K .......... .......... .......... .......... .......... 4% 122M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 3950K .......... .......... .......... .......... .......... 4% 95.1M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4000K .......... .......... .......... .......... .......... 4% 76.6M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4050K .......... .......... .......... .......... .......... 4% 56.7M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4100K .......... .......... .......... .......... .......... 4% 81.9M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4150K .......... .......... .......... .......... .......... 4% 120M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4200K .......... .......... .......... .......... .......... 4% 16.8M 3s 2023-12-03T11:57:25.599 INFO:teuthology.orchestra.run.smithi116.stderr: 4250K .......... .......... .......... .......... .......... 4% 55.5M 3s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4300K .......... .......... .......... .......... .......... 4% 218M 3s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4350K .......... .......... .......... .......... .......... 4% 254M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4400K .......... .......... .......... .......... .......... 4% 222M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4450K .......... .......... .......... .......... .......... 4% 224M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4500K .......... .......... .......... .......... .......... 4% 256M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4550K .......... .......... .......... .......... .......... 4% 234M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4600K .......... .......... .......... .......... .......... 5% 137M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4650K .......... .......... .......... .......... .......... 5% 79.1M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4700K .......... .......... .......... .......... .......... 5% 116M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4750K .......... .......... .......... .......... .......... 5% 112M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4800K .......... .......... .......... .......... .......... 5% 124M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4850K .......... .......... .......... .......... .......... 5% 86.2M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4900K .......... .......... .......... .......... .......... 5% 109M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 4950K .......... .......... .......... .......... .......... 5% 117M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 5000K .......... .......... .......... .......... .......... 5% 121M 2s 2023-12-03T11:57:25.600 INFO:teuthology.orchestra.run.smithi116.stderr: 5050K .......... .......... .......... .......... .......... 5% 51.5M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5100K .......... .......... .......... .......... .......... 5% 117M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5150K .......... .......... .......... .......... .......... 5% 115M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5200K .......... .......... .......... .......... .......... 5% 119M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5250K .......... .......... .......... .......... .......... 5% 89.6M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5300K .......... .......... .......... .......... .......... 5% 109M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5350K .......... .......... .......... .......... .......... 5% 116M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5400K .......... .......... .......... .......... .......... 5% 122M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5450K .......... .......... .......... .......... .......... 5% 118M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5500K .......... .......... .......... .......... .......... 5% 104M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5550K .......... .......... .......... .......... .......... 6% 112M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5600K .......... .......... .......... .......... .......... 6% 121M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5650K .......... .......... .......... .......... .......... 6% 86.0M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5700K .......... .......... .......... .......... .......... 6% 115M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5750K .......... .......... .......... .......... .......... 6% 117M 2s 2023-12-03T11:57:25.601 INFO:teuthology.orchestra.run.smithi116.stderr: 5800K .......... .......... .......... .......... .......... 6% 101M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 5850K .......... .......... .......... .......... .......... 6% 134M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 5900K .......... .......... .......... .......... .......... 6% 114M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 5950K .......... .......... .......... .......... .......... 6% 112M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6000K .......... .......... .......... .......... .......... 6% 124M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6050K .......... .......... .......... .......... .......... 6% 87.6M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6100K .......... .......... .......... .......... .......... 6% 118M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6150K .......... .......... .......... .......... .......... 6% 117M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6200K .......... .......... .......... .......... .......... 6% 119M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6250K .......... .......... .......... .......... .......... 6% 111M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6300K .......... .......... .......... .......... .......... 6% 112M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6350K .......... .......... .......... .......... .......... 6% 120M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6400K .......... .......... .......... .......... .......... 6% 112M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6450K .......... .......... .......... .......... .......... 6% 90.5M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6500K .......... .......... .......... .......... .......... 7% 116M 2s 2023-12-03T11:57:25.602 INFO:teuthology.orchestra.run.smithi116.stderr: 6550K .......... .......... .......... .......... .......... 7% 122M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6600K .......... .......... .......... .......... .......... 7% 93.4M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6650K .......... .......... .......... .......... .......... 7% 79.4M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6700K .......... .......... .......... .......... .......... 7% 118M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6750K .......... .......... .......... .......... .......... 7% 110M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6800K .......... .......... .......... .......... .......... 7% 121M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6850K .......... .......... .......... .......... .......... 7% 89.3M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6900K .......... .......... .......... .......... .......... 7% 117M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 6950K .......... .......... .......... .......... .......... 7% 118M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7000K .......... .......... .......... .......... .......... 7% 111M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7050K .......... .......... .......... .......... .......... 7% 115M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7100K .......... .......... .......... .......... .......... 7% 116M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7150K .......... .......... .......... .......... .......... 7% 119M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7200K .......... .......... .......... .......... .......... 7% 110M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7250K .......... .......... .......... .......... .......... 7% 89.4M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7300K .......... .......... .......... .......... .......... 7% 113M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7350K .......... .......... .......... .......... .......... 7% 112M 2s 2023-12-03T11:57:25.603 INFO:teuthology.orchestra.run.smithi116.stderr: 7400K .......... .......... .......... .......... .......... 8% 114M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7450K .......... .......... .......... .......... .......... 8% 115M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7500K .......... .......... .......... .......... .......... 8% 119M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7550K .......... .......... .......... .......... .......... 8% 114M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7600K .......... .......... .......... .......... .......... 8% 118M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7650K .......... .......... .......... .......... .......... 8% 87.2M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7700K .......... .......... .......... .......... .......... 8% 117M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7750K .......... .......... .......... .......... .......... 8% 117M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7800K .......... .......... .......... .......... .......... 8% 101M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7850K .......... .......... .......... .......... .......... 8% 99.4M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7900K .......... .......... .......... .......... .......... 8% 110M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 7950K .......... .......... .......... .......... .......... 8% 70.3M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8000K .......... .......... .......... .......... .......... 8% 60.6M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8050K .......... .......... .......... .......... .......... 8% 63.0M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8100K .......... .......... .......... .......... .......... 8% 28.0M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8150K .......... .......... .......... .......... .......... 8% 127M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8200K .......... .......... .......... .......... .......... 8% 103M 2s 2023-12-03T11:57:25.604 INFO:teuthology.orchestra.run.smithi116.stderr: 8250K .......... .......... .......... .......... .......... 8% 113M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8300K .......... .......... .......... .......... .......... 8% 120M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8350K .......... .......... .......... .......... .......... 9% 112M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8400K .......... .......... .......... .......... .......... 9% 64.7M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8450K .......... .......... .......... .......... .......... 9% 87.0M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8500K .......... .......... .......... .......... .......... 9% 106M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8550K .......... .......... .......... .......... .......... 9% 95.1M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8600K .......... .......... .......... .......... .......... 9% 115M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8650K .......... .......... .......... .......... .......... 9% 109M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8700K .......... .......... .......... .......... .......... 9% 114M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8750K .......... .......... .......... .......... .......... 9% 118M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8800K .......... .......... .......... .......... .......... 9% 113M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8850K .......... .......... .......... .......... .......... 9% 87.1M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8900K .......... .......... .......... .......... .......... 9% 111M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 8950K .......... .......... .......... .......... .......... 9% 122M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 9000K .......... .......... .......... .......... .......... 9% 59.6M 2s 2023-12-03T11:57:25.605 INFO:teuthology.orchestra.run.smithi116.stderr: 9050K .......... .......... .......... .......... .......... 9% 107M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9100K .......... .......... .......... .......... .......... 9% 114M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9150K .......... .......... .......... .......... .......... 9% 107M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9200K .......... .......... .......... .......... .......... 9% 119M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9250K .......... .......... .......... .......... .......... 10% 82.6M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9300K .......... .......... .......... .......... .......... 10% 121M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9350K .......... .......... .......... .......... .......... 10% 119M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9400K .......... .......... .......... .......... .......... 10% 89.6M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9450K .......... .......... .......... .......... .......... 10% 103M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9500K .......... .......... .......... .......... .......... 10% 114M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9550K .......... .......... .......... .......... .......... 10% 115M 2s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9600K .......... .......... .......... .......... .......... 10% 117M 1s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9650K .......... .......... .......... .......... .......... 10% 86.5M 1s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9700K .......... .......... .......... .......... .......... 10% 113M 1s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9750K .......... .......... .......... .......... .......... 10% 120M 1s 2023-12-03T11:57:25.606 INFO:teuthology.orchestra.run.smithi116.stderr: 9800K .......... .......... .......... .......... .......... 10% 111M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 9850K .......... .......... .......... .......... .......... 10% 107M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 9900K .......... .......... .......... .......... .......... 10% 116M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 9950K .......... .......... .......... .......... .......... 10% 118M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10000K .......... .......... .......... .......... .......... 10% 111M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10050K .......... .......... .......... .......... .......... 10% 90.1M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10100K .......... .......... .......... .......... .......... 10% 109M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10150K .......... .......... .......... .......... .......... 10% 116M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10200K .......... .......... .......... .......... .......... 11% 111M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10250K .......... .......... .......... .......... .......... 11% 120M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10300K .......... .......... .......... .......... .......... 11% 111M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10350K .......... .......... .......... .......... .......... 11% 115M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10400K .......... .......... .......... .......... .......... 11% 115M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10450K .......... .......... .......... .......... .......... 11% 89.4M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10500K .......... .......... .......... .......... .......... 11% 106M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10550K .......... .......... .......... .......... .......... 11% 90.8M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10600K .......... .......... .......... .......... .......... 11% 88.8M 1s 2023-12-03T11:57:25.607 INFO:teuthology.orchestra.run.smithi116.stderr: 10650K .......... .......... .......... .......... .......... 11% 110M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10700K .......... .......... .......... .......... .......... 11% 120M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10750K .......... .......... .......... .......... .......... 11% 112M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10800K .......... .......... .......... .......... .......... 11% 116M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10850K .......... .......... .......... .......... .......... 11% 90.1M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10900K .......... .......... .......... .......... .......... 11% 115M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 10950K .......... .......... .......... .......... .......... 11% 116M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11000K .......... .......... .......... .......... .......... 11% 115M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11050K .......... .......... .......... .......... .......... 11% 103M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11100K .......... .......... .......... .......... .......... 12% 124M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11150K .......... .......... .......... .......... .......... 12% 118M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11200K .......... .......... .......... .......... .......... 12% 116M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11250K .......... .......... .......... .......... .......... 12% 88.7M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11300K .......... .......... .......... .......... .......... 12% 118M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11350K .......... .......... .......... .......... .......... 12% 114M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11400K .......... .......... .......... .......... .......... 12% 119M 1s 2023-12-03T11:57:25.608 INFO:teuthology.orchestra.run.smithi116.stderr: 11450K .......... .......... .......... .......... .......... 12% 107M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11500K .......... .......... .......... .......... .......... 12% 113M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11550K .......... .......... .......... .......... .......... 12% 119M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11600K .......... .......... .......... .......... .......... 12% 114M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11650K .......... .......... .......... .......... .......... 12% 71.9M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11700K .......... .......... .......... .......... .......... 12% 65.4M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11750K .......... .......... .......... .......... .......... 12% 113M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11800K .......... .......... .......... .......... .......... 12% 119M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11850K .......... .......... .......... .......... .......... 12% 112M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11900K .......... .......... .......... .......... .......... 12% 66.6M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 11950K .......... .......... .......... .......... .......... 12% 59.5M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 12000K .......... .......... .......... .......... .......... 12% 70.6M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 12050K .......... .......... .......... .......... .......... 13% 53.0M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 12100K .......... .......... .......... .......... .......... 13% 116M 1s 2023-12-03T11:57:25.609 INFO:teuthology.orchestra.run.smithi116.stderr: 12150K .......... .......... .......... .......... .......... 13% 90.9M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12200K .......... .......... .......... .......... .......... 13% 147M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12250K .......... .......... .......... .......... .......... 13% 110M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12300K .......... .......... .......... .......... .......... 13% 117M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12350K .......... .......... .......... .......... .......... 13% 89.5M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12400K .......... .......... .......... .......... .......... 13% 115M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12450K .......... .......... .......... .......... .......... 13% 90.2M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12500K .......... .......... .......... .......... .......... 13% 82.8M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12550K .......... .......... .......... .......... .......... 13% 119M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12600K .......... .......... .......... .......... .......... 13% 111M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12650K .......... .......... .......... .......... .......... 13% 112M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12700K .......... .......... .......... .......... .......... 13% 110M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12750K .......... .......... .......... .......... .......... 13% 118M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12800K .......... .......... .......... .......... .......... 13% 94.9M 1s 2023-12-03T11:57:25.610 INFO:teuthology.orchestra.run.smithi116.stderr: 12850K .......... .......... .......... .......... .......... 13% 87.4M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 12900K .......... .......... .......... .......... .......... 13% 115M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 12950K .......... .......... .......... .......... .......... 13% 60.1M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13000K .......... .......... .......... .......... .......... 14% 124M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13050K .......... .......... .......... .......... .......... 14% 106M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13100K .......... .......... .......... .......... .......... 14% 105M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13150K .......... .......... .......... .......... .......... 14% 114M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13200K .......... .......... .......... .......... .......... 14% 118M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13250K .......... .......... .......... .......... .......... 14% 87.3M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13300K .......... .......... .......... .......... .......... 14% 115M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13350K .......... .......... .......... .......... .......... 14% 118M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13400K .......... .......... .......... .......... .......... 14% 112M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13450K .......... .......... .......... .......... .......... 14% 111M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13500K .......... .......... .......... .......... .......... 14% 114M 1s 2023-12-03T11:57:25.611 INFO:teuthology.orchestra.run.smithi116.stderr: 13550K .......... .......... .......... .......... .......... 14% 115M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13600K .......... .......... .......... .......... .......... 14% 115M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13650K .......... .......... .......... .......... .......... 14% 89.2M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13700K .......... .......... .......... .......... .......... 14% 121M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13750K .......... .......... .......... .......... .......... 14% 112M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13800K .......... .......... .......... .......... .......... 14% 121M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13850K .......... .......... .......... .......... .......... 14% 107M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13900K .......... .......... .......... .......... .......... 15% 115M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 13950K .......... .......... .......... .......... .......... 15% 115M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14000K .......... .......... .......... .......... .......... 15% 118M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14050K .......... .......... .......... .......... .......... 15% 87.3M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14100K .......... .......... .......... .......... .......... 15% 68.2M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14150K .......... .......... .......... .......... .......... 15% 78.1M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14200K .......... .......... .......... .......... .......... 15% 112M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14250K .......... .......... .......... .......... .......... 15% 109M 1s 2023-12-03T11:57:25.612 INFO:teuthology.orchestra.run.smithi116.stderr: 14300K .......... .......... .......... .......... .......... 15% 110M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14350K .......... .......... .......... .......... .......... 15% 121M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14400K .......... .......... .......... .......... .......... 15% 115M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14450K .......... .......... .......... .......... .......... 15% 88.7M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14500K .......... .......... .......... .......... .......... 15% 117M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14550K .......... .......... .......... .......... .......... 15% 75.7M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14600K .......... .......... .......... .......... .......... 15% 119M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14650K .......... .......... .......... .......... .......... 15% 110M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14700K .......... .......... .......... .......... .......... 15% 116M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14750K .......... .......... .......... .......... .......... 15% 114M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14800K .......... .......... .......... .......... .......... 15% 116M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14850K .......... .......... .......... .......... .......... 16% 88.0M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14900K .......... .......... .......... .......... .......... 16% 119M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 14950K .......... .......... .......... .......... .......... 16% 108M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 15000K .......... .......... .......... .......... .......... 16% 109M 1s 2023-12-03T11:57:25.613 INFO:teuthology.orchestra.run.smithi116.stderr: 15050K .......... .......... .......... .......... .......... 16% 113M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15100K .......... .......... .......... .......... .......... 16% 117M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15150K .......... .......... .......... .......... .......... 16% 117M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15200K .......... .......... .......... .......... .......... 16% 114M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15250K .......... .......... .......... .......... .......... 16% 89.5M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15300K .......... .......... .......... .......... .......... 16% 119M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15350K .......... .......... .......... .......... .......... 16% 115M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15400K .......... .......... .......... .......... .......... 16% 115M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15450K .......... .......... .......... .......... .......... 16% 106M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15500K .......... .......... .......... .......... .......... 16% 119M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15550K .......... .......... .......... .......... .......... 16% 116M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15600K .......... .......... .......... .......... .......... 16% 112M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15650K .......... .......... .......... .......... .......... 16% 92.0M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15700K .......... .......... .......... .......... .......... 16% 104M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15750K .......... .......... .......... .......... .......... 17% 123M 1s 2023-12-03T11:57:25.614 INFO:teuthology.orchestra.run.smithi116.stderr: 15800K .......... .......... .......... .......... .......... 17% 66.4M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 15850K .......... .......... .......... .......... .......... 17% 111M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 15900K .......... .......... .......... .......... .......... 17% 61.7M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 15950K .......... .......... .......... .......... .......... 17% 56.9M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16000K .......... .......... .......... .......... .......... 17% 66.9M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16050K .......... .......... .......... .......... .......... 17% 103M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16100K .......... .......... .......... .......... .......... 17% 104M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16150K .......... .......... .......... .......... .......... 17% 108M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16200K .......... .......... .......... .......... .......... 17% 122M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16250K .......... .......... .......... .......... .......... 17% 88.0M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16300K .......... .......... .......... .......... .......... 17% 139M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16350K .......... .......... .......... .......... .......... 17% 92.3M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16400K .......... .......... .......... .......... .......... 17% 136M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16450K .......... .......... .......... .......... .......... 17% 71.8M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16500K .......... .......... .......... .......... .......... 17% 115M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16550K .......... .......... .......... .......... .......... 17% 98.9M 1s 2023-12-03T11:57:25.615 INFO:teuthology.orchestra.run.smithi116.stderr: 16600K .......... .......... .......... .......... .......... 17% 147M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16650K .......... .......... .......... .......... .......... 17% 109M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16700K .......... .......... .......... .......... .......... 18% 115M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16750K .......... .......... .......... .......... .......... 18% 116M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16800K .......... .......... .......... .......... .......... 18% 115M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16850K .......... .......... .......... .......... .......... 18% 90.1M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16900K .......... .......... .......... .......... .......... 18% 64.8M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 16950K .......... .......... .......... .......... .......... 18% 77.7M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17000K .......... .......... .......... .......... .......... 18% 118M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17050K .......... .......... .......... .......... .......... 18% 109M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17100K .......... .......... .......... .......... .......... 18% 123M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17150K .......... .......... .......... .......... .......... 18% 109M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17200K .......... .......... .......... .......... .......... 18% 119M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17250K .......... .......... .......... .......... .......... 18% 87.0M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17300K .......... .......... .......... .......... .......... 18% 114M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17350K .......... .......... .......... .......... .......... 18% 117M 1s 2023-12-03T11:57:25.616 INFO:teuthology.orchestra.run.smithi116.stderr: 17400K .......... .......... .......... .......... .......... 18% 119M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17450K .......... .......... .......... .......... .......... 18% 110M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17500K .......... .......... .......... .......... .......... 18% 114M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17550K .......... .......... .......... .......... .......... 18% 117M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17600K .......... .......... .......... .......... .......... 18% 112M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17650K .......... .......... .......... .......... .......... 19% 90.1M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17700K .......... .......... .......... .......... .......... 19% 121M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17750K .......... .......... .......... .......... .......... 19% 111M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17800K .......... .......... .......... .......... .......... 19% 118M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17850K .......... .......... .......... .......... .......... 19% 110M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17900K .......... .......... .......... .......... .......... 19% 116M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 17950K .......... .......... .......... .......... .......... 19% 120M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 18000K .......... .......... .......... .......... .......... 19% 117M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 18050K .......... .......... .......... .......... .......... 19% 87.8M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 18100K .......... .......... .......... .......... .......... 19% 113M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 18150K .......... .......... .......... .......... .......... 19% 118M 1s 2023-12-03T11:57:25.617 INFO:teuthology.orchestra.run.smithi116.stderr: 18200K .......... .......... .......... .......... .......... 19% 107M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18250K .......... .......... .......... .......... .......... 19% 119M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18300K .......... .......... .......... .......... .......... 19% 74.3M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18350K .......... .......... .......... .......... .......... 19% 113M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18400K .......... .......... .......... .......... .......... 19% 50.1M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18450K .......... .......... .......... .......... .......... 19% 66.0M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18500K .......... .......... .......... .......... .......... 19% 90.7M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18550K .......... .......... .......... .......... .......... 20% 112M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18600K .......... .......... .......... .......... .......... 20% 115M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18650K .......... .......... .......... .......... .......... 20% 103M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18700K .......... .......... .......... .......... .......... 20% 122M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18750K .......... .......... .......... .......... .......... 20% 126M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18800K .......... .......... .......... .......... .......... 20% 110M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18850K .......... .......... .......... .......... .......... 20% 90.2M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18900K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.618 INFO:teuthology.orchestra.run.smithi116.stderr: 18950K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19000K .......... .......... .......... .......... .......... 20% 110M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19050K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19100K .......... .......... .......... .......... .......... 20% 111M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19150K .......... .......... .......... .......... .......... 20% 119M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19200K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19250K .......... .......... .......... .......... .......... 20% 82.6M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19300K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19350K .......... .......... .......... .......... .......... 20% 116M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19400K .......... .......... .......... .......... .......... 20% 115M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19450K .......... .......... .......... .......... .......... 20% 108M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19500K .......... .......... .......... .......... .......... 21% 121M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19550K .......... .......... .......... .......... .......... 21% 112M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19600K .......... .......... .......... .......... .......... 21% 117M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19650K .......... .......... .......... .......... .......... 21% 88.4M 1s 2023-12-03T11:57:25.619 INFO:teuthology.orchestra.run.smithi116.stderr: 19700K .......... .......... .......... .......... .......... 21% 115M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 19750K .......... .......... .......... .......... .......... 21% 75.0M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 19800K .......... .......... .......... .......... .......... 21% 119M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 19850K .......... .......... .......... .......... .......... 21% 57.6M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 19900K .......... .......... .......... .......... .......... 21% 47.1M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 19950K .......... .......... .......... .......... .......... 21% 76.8M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20000K .......... .......... .......... .......... .......... 21% 110M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20050K .......... .......... .......... .......... .......... 21% 93.2M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20100K .......... .......... .......... .......... .......... 21% 112M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20150K .......... .......... .......... .......... .......... 21% 103M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20200K .......... .......... .......... .......... .......... 21% 98.4M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20250K .......... .......... .......... .......... .......... 21% 115M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20300K .......... .......... .......... .......... .......... 21% 113M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20350K .......... .......... .......... .......... .......... 21% 117M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20400K .......... .......... .......... .......... .......... 22% 82.7M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20450K .......... .......... .......... .......... .......... 22% 94.4M 1s 2023-12-03T11:57:25.620 INFO:teuthology.orchestra.run.smithi116.stderr: 20500K .......... .......... .......... .......... .......... 22% 114M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20550K .......... .......... .......... .......... .......... 22% 112M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20600K .......... .......... .......... .......... .......... 22% 113M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20650K .......... .......... .......... .......... .......... 22% 114M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20700K .......... .......... .......... .......... .......... 22% 114M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20750K .......... .......... .......... .......... .......... 22% 115M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20800K .......... .......... .......... .......... .......... 22% 116M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20850K .......... .......... .......... .......... .......... 22% 73.4M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20900K .......... .......... .......... .......... .......... 22% 74.6M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 20950K .......... .......... .......... .......... .......... 22% 90.7M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21000K .......... .......... .......... .......... .......... 22% 124M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21050K .......... .......... .......... .......... .......... 22% 102M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21100K .......... .......... .......... .......... .......... 22% 113M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21150K .......... .......... .......... .......... .......... 22% 116M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21200K .......... .......... .......... .......... .......... 22% 122M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21250K .......... .......... .......... .......... .......... 22% 87.3M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21300K .......... .......... .......... .......... .......... 22% 122M 1s 2023-12-03T11:57:25.621 INFO:teuthology.orchestra.run.smithi116.stderr: 21350K .......... .......... .......... .......... .......... 23% 115M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21400K .......... .......... .......... .......... .......... 23% 117M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21450K .......... .......... .......... .......... .......... 23% 110M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21500K .......... .......... .......... .......... .......... 23% 113M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21550K .......... .......... .......... .......... .......... 23% 115M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21600K .......... .......... .......... .......... .......... 23% 105M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21650K .......... .......... .......... .......... .......... 23% 87.3M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21700K .......... .......... .......... .......... .......... 23% 125M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21750K .......... .......... .......... .......... .......... 23% 111M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21800K .......... .......... .......... .......... .......... 23% 112M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21850K .......... .......... .......... .......... .......... 23% 111M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21900K .......... .......... .......... .......... .......... 23% 125M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 21950K .......... .......... .......... .......... .......... 23% 109M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 22000K .......... .......... .......... .......... .......... 23% 114M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 22050K .......... .......... .......... .......... .......... 23% 89.1M 1s 2023-12-03T11:57:25.622 INFO:teuthology.orchestra.run.smithi116.stderr: 22100K .......... .......... .......... .......... .......... 23% 117M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22150K .......... .......... .......... .......... .......... 23% 116M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22200K .......... .......... .......... .......... .......... 23% 116M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22250K .......... .......... .......... .......... .......... 24% 112M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22300K .......... .......... .......... .......... .......... 24% 114M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22350K .......... .......... .......... .......... .......... 24% 117M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22400K .......... .......... .......... .......... .......... 24% 112M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22450K .......... .......... .......... .......... .......... 24% 85.1M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22500K .......... .......... .......... .......... .......... 24% 122M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22550K .......... .......... .......... .......... .......... 24% 112M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22600K .......... .......... .......... .......... .......... 24% 121M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22650K .......... .......... .......... .......... .......... 24% 109M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22700K .......... .......... .......... .......... .......... 24% 112M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22750K .......... .......... .......... .......... .......... 24% 121M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22800K .......... .......... .......... .......... .......... 24% 114M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22850K .......... .......... .......... .......... .......... 24% 82.9M 1s 2023-12-03T11:57:25.623 INFO:teuthology.orchestra.run.smithi116.stderr: 22900K .......... .......... .......... .......... .......... 24% 135M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 22950K .......... .......... .......... .......... .......... 24% 115M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23000K .......... .......... .......... .......... .......... 24% 108M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23050K .......... .......... .......... .......... .......... 24% 108M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23100K .......... .......... .......... .......... .......... 24% 113M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23150K .......... .......... .......... .......... .......... 24% 93.6M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23200K .......... .......... .......... .......... .......... 25% 114M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23250K .......... .......... .......... .......... .......... 25% 81.7M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23300K .......... .......... .......... .......... .......... 25% 107M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23350K .......... .......... .......... .......... .......... 25% 149M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23400K .......... .......... .......... .......... .......... 25% 113M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23450K .......... .......... .......... .......... .......... 25% 105M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23500K .......... .......... .......... .......... .......... 25% 116M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23550K .......... .......... .......... .......... .......... 25% 125M 1s 2023-12-03T11:57:25.624 INFO:teuthology.orchestra.run.smithi116.stderr: 23600K .......... .......... .......... .......... .......... 25% 114M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23650K .......... .......... .......... .......... .......... 25% 90.3M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23700K .......... .......... .......... .......... .......... 25% 110M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23750K .......... .......... .......... .......... .......... 25% 117M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23800K .......... .......... .......... .......... .......... 25% 86.3M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23850K .......... .......... .......... .......... .......... 25% 116M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23900K .......... .......... .......... .......... .......... 25% 113M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 23950K .......... .......... .......... .......... .......... 25% 119M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24000K .......... .......... .......... .......... .......... 25% 114M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24050K .......... .......... .......... .......... .......... 25% 89.5M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24100K .......... .......... .......... .......... .......... 25% 121M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24150K .......... .......... .......... .......... .......... 26% 105M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24200K .......... .......... .......... .......... .......... 26% 110M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24250K .......... .......... .......... .......... .......... 26% 101M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24300K .......... .......... .......... .......... .......... 26% 130M 1s 2023-12-03T11:57:25.625 INFO:teuthology.orchestra.run.smithi116.stderr: 24350K .......... .......... .......... .......... .......... 26% 131M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24400K .......... .......... .......... .......... .......... 26% 114M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24450K .......... .......... .......... .......... .......... 26% 88.6M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24500K .......... .......... .......... .......... .......... 26% 115M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24550K .......... .......... .......... .......... .......... 26% 114M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24600K .......... .......... .......... .......... .......... 26% 115M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24650K .......... .......... .......... .......... .......... 26% 72.8M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24700K .......... .......... .......... .......... .......... 26% 115M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24750K .......... .......... .......... .......... .......... 26% 117M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24800K .......... .......... .......... .......... .......... 26% 96.2M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24850K .......... .......... .......... .......... .......... 26% 66.6M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24900K .......... .......... .......... .......... .......... 26% 110M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 24950K .......... .......... .......... .......... .......... 26% 113M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 25000K .......... .......... .......... .......... .......... 26% 84.7M 1s 2023-12-03T11:57:25.626 INFO:teuthology.orchestra.run.smithi116.stderr: 25050K .......... .......... .......... .......... .......... 27% 44.8M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25100K .......... .......... .......... .......... .......... 27% 115M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25150K .......... .......... .......... .......... .......... 27% 112M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25200K .......... .......... .......... .......... .......... 27% 118M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25250K .......... .......... .......... .......... .......... 27% 86.7M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25300K .......... .......... .......... .......... .......... 27% 98.0M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25350K .......... .......... .......... .......... .......... 27% 117M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25400K .......... .......... .......... .......... .......... 27% 118M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25450K .......... .......... .......... .......... .......... 27% 108M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25500K .......... .......... .......... .......... .......... 27% 110M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25550K .......... .......... .......... .......... .......... 27% 119M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25600K .......... .......... .......... .......... .......... 27% 117M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25650K .......... .......... .......... .......... .......... 27% 87.8M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25700K .......... .......... .......... .......... .......... 27% 110M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25750K .......... .......... .......... .......... .......... 27% 112M 1s 2023-12-03T11:57:25.627 INFO:teuthology.orchestra.run.smithi116.stderr: 25800K .......... .......... .......... .......... .......... 27% 117M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 25850K .......... .......... .......... .......... .......... 27% 108M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 25900K .......... .......... .......... .......... .......... 27% 117M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 25950K .......... .......... .......... .......... .......... 27% 114M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26000K .......... .......... .......... .......... .......... 28% 107M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26050K .......... .......... .......... .......... .......... 28% 90.3M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26100K .......... .......... .......... .......... .......... 28% 115M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26150K .......... .......... .......... .......... .......... 28% 86.4M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26200K .......... .......... .......... .......... .......... 28% 119M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26250K .......... .......... .......... .......... .......... 28% 110M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26300K .......... .......... .......... .......... .......... 28% 116M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26350K .......... .......... .......... .......... .......... 28% 114M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26400K .......... .......... .......... .......... .......... 28% 113M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26450K .......... .......... .......... .......... .......... 28% 91.3M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26500K .......... .......... .......... .......... .......... 28% 110M 1s 2023-12-03T11:57:25.628 INFO:teuthology.orchestra.run.smithi116.stderr: 26550K .......... .......... .......... .......... .......... 28% 117M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26600K .......... .......... .......... .......... .......... 28% 117M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26650K .......... .......... .......... .......... .......... 28% 110M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26700K .......... .......... .......... .......... .......... 28% 89.2M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26750K .......... .......... .......... .......... .......... 28% 113M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26800K .......... .......... .......... .......... .......... 28% 116M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26850K .......... .......... .......... .......... .......... 28% 91.8M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26900K .......... .......... .......... .......... .......... 29% 116M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 26950K .......... .......... .......... .......... .......... 29% 117M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27000K .......... .......... .......... .......... .......... 29% 115M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27050K .......... .......... .......... .......... .......... 29% 115M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27100K .......... .......... .......... .......... .......... 29% 98.8M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27150K .......... .......... .......... .......... .......... 29% 104M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27200K .......... .......... .......... .......... .......... 29% 116M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27250K .......... .......... .......... .......... .......... 29% 91.7M 1s 2023-12-03T11:57:25.629 INFO:teuthology.orchestra.run.smithi116.stderr: 27300K .......... .......... .......... .......... .......... 29% 105M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27350K .......... .......... .......... .......... .......... 29% 117M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27400K .......... .......... .......... .......... .......... 29% 125M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27450K .......... .......... .......... .......... .......... 29% 111M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27500K .......... .......... .......... .......... .......... 29% 117M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27550K .......... .......... .......... .......... .......... 29% 94.2M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27600K .......... .......... .......... .......... .......... 29% 139M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27650K .......... .......... .......... .......... .......... 29% 80.3M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27700K .......... .......... .......... .......... .......... 29% 130M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27750K .......... .......... .......... .......... .......... 29% 121M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27800K .......... .......... .......... .......... .......... 29% 107M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27850K .......... .......... .......... .......... .......... 30% 92.2M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27900K .......... .......... .......... .......... .......... 30% 126M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 27950K .......... .......... .......... .......... .......... 30% 149M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 28000K .......... .......... .......... .......... .......... 30% 106M 1s 2023-12-03T11:57:25.630 INFO:teuthology.orchestra.run.smithi116.stderr: 28050K .......... .......... .......... .......... .......... 30% 90.7M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28100K .......... .......... .......... .......... .......... 30% 115M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28150K .......... .......... .......... .......... .......... 30% 117M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28200K .......... .......... .......... .......... .......... 30% 114M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28250K .......... .......... .......... .......... .......... 30% 78.2M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28300K .......... .......... .......... .......... .......... 30% 118M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28350K .......... .......... .......... .......... .......... 30% 82.7M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28400K .......... .......... .......... .......... .......... 30% 202M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28450K .......... .......... .......... .......... .......... 30% 89.3M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28500K .......... .......... .......... .......... .......... 30% 116M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28550K .......... .......... .......... .......... .......... 30% 111M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28600K .......... .......... .......... .......... .......... 30% 112M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28650K .......... .......... .......... .......... .......... 30% 117M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28700K .......... .......... .......... .......... .......... 30% 113M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28750K .......... .......... .......... .......... .......... 31% 117M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28800K .......... .......... .......... .......... .......... 31% 112M 1s 2023-12-03T11:57:25.631 INFO:teuthology.orchestra.run.smithi116.stderr: 28850K .......... .......... .......... .......... .......... 31% 91.5M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 28900K .......... .......... .......... .......... .......... 31% 115M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 28950K .......... .......... .......... .......... .......... 31% 114M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29000K .......... .......... .......... .......... .......... 31% 113M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29050K .......... .......... .......... .......... .......... 31% 110M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29100K .......... .......... .......... .......... .......... 31% 122M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29150K .......... .......... .......... .......... .......... 31% 113M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29200K .......... .......... .......... .......... .......... 31% 113M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29250K .......... .......... .......... .......... .......... 31% 82.6M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29300K .......... .......... .......... .......... .......... 31% 124M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29350K .......... .......... .......... .......... .......... 31% 116M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29400K .......... .......... .......... .......... .......... 31% 113M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29450K .......... .......... .......... .......... .......... 31% 115M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29500K .......... .......... .......... .......... .......... 31% 109M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29550K .......... .......... .......... .......... .......... 31% 63.2M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29600K .......... .......... .......... .......... .......... 31% 101M 1s 2023-12-03T11:57:25.632 INFO:teuthology.orchestra.run.smithi116.stderr: 29650K .......... .......... .......... .......... .......... 31% 67.5M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29700K .......... .......... .......... .......... .......... 32% 91.5M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29750K .......... .......... .......... .......... .......... 32% 107M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29800K .......... .......... .......... .......... .......... 32% 116M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29850K .......... .......... .......... .......... .......... 32% 109M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29900K .......... .......... .......... .......... .......... 32% 116M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 29950K .......... .......... .......... .......... .......... 32% 117M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30000K .......... .......... .......... .......... .......... 32% 106M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30050K .......... .......... .......... .......... .......... 32% 92.2M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30100K .......... .......... .......... .......... .......... 32% 111M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30150K .......... .......... .......... .......... .......... 32% 113M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30200K .......... .......... .......... .......... .......... 32% 116M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30250K .......... .......... .......... .......... .......... 32% 110M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30300K .......... .......... .......... .......... .......... 32% 115M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30350K .......... .......... .......... .......... .......... 32% 116M 1s 2023-12-03T11:57:25.633 INFO:teuthology.orchestra.run.smithi116.stderr: 30400K .......... .......... .......... .......... .......... 32% 102M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30450K .......... .......... .......... .......... .......... 32% 84.1M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30500K .......... .......... .......... .......... .......... 32% 141M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30550K .......... .......... .......... .......... .......... 32% 120M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30600K .......... .......... .......... .......... .......... 32% 116M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30650K .......... .......... .......... .......... .......... 33% 101M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30700K .......... .......... .......... .......... .......... 33% 113M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30750K .......... .......... .......... .......... .......... 33% 117M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30800K .......... .......... .......... .......... .......... 33% 121M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30850K .......... .......... .......... .......... .......... 33% 72.1M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30900K .......... .......... .......... .......... .......... 33% 109M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 30950K .......... .......... .......... .......... .......... 33% 120M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 31000K .......... .......... .......... .......... .......... 33% 118M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 31050K .......... .......... .......... .......... .......... 33% 107M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 31100K .......... .......... .......... .......... .......... 33% 106M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 31150K .......... .......... .......... .......... .......... 33% 123M 1s 2023-12-03T11:57:25.634 INFO:teuthology.orchestra.run.smithi116.stderr: 31200K .......... .......... .......... .......... .......... 33% 118M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31250K .......... .......... .......... .......... .......... 33% 73.7M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31300K .......... .......... .......... .......... .......... 33% 121M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31350K .......... .......... .......... .......... .......... 33% 112M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31400K .......... .......... .......... .......... .......... 33% 116M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31450K .......... .......... .......... .......... .......... 33% 111M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31500K .......... .......... .......... .......... .......... 33% 115M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31550K .......... .......... .......... .......... .......... 34% 111M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31600K .......... .......... .......... .......... .......... 34% 116M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31650K .......... .......... .......... .......... .......... 34% 89.0M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31700K .......... .......... .......... .......... .......... 34% 117M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31750K .......... .......... .......... .......... .......... 34% 118M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31800K .......... .......... .......... .......... .......... 34% 113M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31850K .......... .......... .......... .......... .......... 34% 110M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31900K .......... .......... .......... .......... .......... 34% 116M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 31950K .......... .......... .......... .......... .......... 34% 119M 1s 2023-12-03T11:57:25.635 INFO:teuthology.orchestra.run.smithi116.stderr: 32000K .......... .......... .......... .......... .......... 34% 116M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32050K .......... .......... .......... .......... .......... 34% 87.4M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32100K .......... .......... .......... .......... .......... 34% 121M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32150K .......... .......... .......... .......... .......... 34% 116M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32200K .......... .......... .......... .......... .......... 34% 115M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32250K .......... .......... .......... .......... .......... 34% 103M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32300K .......... .......... .......... .......... .......... 34% 127M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32350K .......... .......... .......... .......... .......... 34% 117M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32400K .......... .......... .......... .......... .......... 34% 117M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32450K .......... .......... .......... .......... .......... 34% 90.8M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32500K .......... .......... .......... .......... .......... 35% 111M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32550K .......... .......... .......... .......... .......... 35% 116M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32600K .......... .......... .......... .......... .......... 35% 115M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32650K .......... .......... .......... .......... .......... 35% 103M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32700K .......... .......... .......... .......... .......... 35% 121M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32750K .......... .......... .......... .......... .......... 35% 127M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32800K .......... .......... .......... .......... .......... 35% 114M 1s 2023-12-03T11:57:25.636 INFO:teuthology.orchestra.run.smithi116.stderr: 32850K .......... .......... .......... .......... .......... 35% 87.8M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 32900K .......... .......... .......... .......... .......... 35% 116M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 32950K .......... .......... .......... .......... .......... 35% 112M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33000K .......... .......... .......... .......... .......... 35% 116M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33050K .......... .......... .......... .......... .......... 35% 107M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33100K .......... .......... .......... .......... .......... 35% 106M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33150K .......... .......... .......... .......... .......... 35% 123M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33200K .......... .......... .......... .......... .......... 35% 111M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33250K .......... .......... .......... .......... .......... 35% 81.4M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33300K .......... .......... .......... .......... .......... 35% 106M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33350K .......... .......... .......... .......... .......... 35% 105M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33400K .......... .......... .......... .......... .......... 36% 144M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33450K .......... .......... .......... .......... .......... 36% 116M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33500K .......... .......... .......... .......... .......... 36% 113M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33550K .......... .......... .......... .......... .......... 36% 119M 1s 2023-12-03T11:57:25.637 INFO:teuthology.orchestra.run.smithi116.stderr: 33600K .......... .......... .......... .......... .......... 36% 111M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33650K .......... .......... .......... .......... .......... 36% 91.3M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33700K .......... .......... .......... .......... .......... 36% 114M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33750K .......... .......... .......... .......... .......... 36% 116M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33800K .......... .......... .......... .......... .......... 36% 113M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33850K .......... .......... .......... .......... .......... 36% 113M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33900K .......... .......... .......... .......... .......... 36% 121M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 33950K .......... .......... .......... .......... .......... 36% 112M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34000K .......... .......... .......... .......... .......... 36% 116M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34050K .......... .......... .......... .......... .......... 36% 92.0M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34100K .......... .......... .......... .......... .......... 36% 109M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34150K .......... .......... .......... .......... .......... 36% 123M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34200K .......... .......... .......... .......... .......... 36% 97.3M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34250K .......... .......... .......... .......... .......... 36% 117M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34300K .......... .......... .......... .......... .......... 36% 113M 1s 2023-12-03T11:57:25.638 INFO:teuthology.orchestra.run.smithi116.stderr: 34350K .......... .......... .......... .......... .......... 37% 118M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34400K .......... .......... .......... .......... .......... 37% 112M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34450K .......... .......... .......... .......... .......... 37% 89.2M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34500K .......... .......... .......... .......... .......... 37% 118M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34550K .......... .......... .......... .......... .......... 37% 94.4M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34600K .......... .......... .......... .......... .......... 37% 92.4M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34650K .......... .......... .......... .......... .......... 37% 105M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34700K .......... .......... .......... .......... .......... 37% 109M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34750K .......... .......... .......... .......... .......... 37% 83.9M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34800K .......... .......... .......... .......... .......... 37% 93.1M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34850K .......... .......... .......... .......... .......... 37% 80.4M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34900K .......... .......... .......... .......... .......... 37% 92.5M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 34950K .......... .......... .......... .......... .......... 37% 114M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 35000K .......... .......... .......... .......... .......... 37% 116M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 35050K .......... .......... .......... .......... .......... 37% 111M 1s 2023-12-03T11:57:25.639 INFO:teuthology.orchestra.run.smithi116.stderr: 35100K .......... .......... .......... .......... .......... 37% 104M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35150K .......... .......... .......... .......... .......... 37% 111M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35200K .......... .......... .......... .......... .......... 37% 113M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35250K .......... .......... .......... .......... .......... 37% 91.0M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35300K .......... .......... .......... .......... .......... 38% 112M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35350K .......... .......... .......... .......... .......... 38% 110M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35400K .......... .......... .......... .......... .......... 38% 113M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35450K .......... .......... .......... .......... .......... 38% 115M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35500K .......... .......... .......... .......... .......... 38% 113M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35550K .......... .......... .......... .......... .......... 38% 111M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35600K .......... .......... .......... .......... .......... 38% 105M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35650K .......... .......... .......... .......... .......... 38% 91.1M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35700K .......... .......... .......... .......... .......... 38% 114M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35750K .......... .......... .......... .......... .......... 38% 117M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35800K .......... .......... .......... .......... .......... 38% 100M 1s 2023-12-03T11:57:25.640 INFO:teuthology.orchestra.run.smithi116.stderr: 35850K .......... .......... .......... .......... .......... 38% 114M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 35900K .......... .......... .......... .......... .......... 38% 118M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 35950K .......... .......... .......... .......... .......... 38% 114M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36000K .......... .......... .......... .......... .......... 38% 95.7M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36050K .......... .......... .......... .......... .......... 38% 83.8M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36100K .......... .......... .......... .......... .......... 38% 117M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36150K .......... .......... .......... .......... .......... 38% 113M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36200K .......... .......... .......... .......... .......... 39% 71.2M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36250K .......... .......... .......... .......... .......... 39% 102M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36300K .......... .......... .......... .......... .......... 39% 127M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36350K .......... .......... .......... .......... .......... 39% 109M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36400K .......... .......... .......... .......... .......... 39% 117M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36450K .......... .......... .......... .......... .......... 39% 92.7M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36500K .......... .......... .......... .......... .......... 39% 109M 1s 2023-12-03T11:57:25.641 INFO:teuthology.orchestra.run.smithi116.stderr: 36550K .......... .......... .......... .......... .......... 39% 115M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36600K .......... .......... .......... .......... .......... 39% 120M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36650K .......... .......... .......... .......... .......... 39% 113M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36700K .......... .......... .......... .......... .......... 39% 114M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36750K .......... .......... .......... .......... .......... 39% 117M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36800K .......... .......... .......... .......... .......... 39% 115M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36850K .......... .......... .......... .......... .......... 39% 85.3M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36900K .......... .......... .......... .......... .......... 39% 118M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 36950K .......... .......... .......... .......... .......... 39% 114M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37000K .......... .......... .......... .......... .......... 39% 119M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37050K .......... .......... .......... .......... .......... 39% 107M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37100K .......... .......... .......... .......... .......... 39% 112M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37150K .......... .......... .......... .......... .......... 40% 112M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37200K .......... .......... .......... .......... .......... 40% 122M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37250K .......... .......... .......... .......... .......... 40% 81.1M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37300K .......... .......... .......... .......... .......... 40% 114M 1s 2023-12-03T11:57:25.642 INFO:teuthology.orchestra.run.smithi116.stderr: 37350K .......... .......... .......... .......... .......... 40% 117M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37400K .......... .......... .......... .......... .......... 40% 116M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37450K .......... .......... .......... .......... .......... 40% 99.3M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37500K .......... .......... .......... .......... .......... 40% 112M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37550K .......... .......... .......... .......... .......... 40% 121M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37600K .......... .......... .......... .......... .......... 40% 110M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37650K .......... .......... .......... .......... .......... 40% 87.5M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37700K .......... .......... .......... .......... .......... 40% 119M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37750K .......... .......... .......... .......... .......... 40% 117M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37800K .......... .......... .......... .......... .......... 40% 116M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37850K .......... .......... .......... .......... .......... 40% 109M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37900K .......... .......... .......... .......... .......... 40% 112M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 37950K .......... .......... .......... .......... .......... 40% 102M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 38000K .......... .......... .......... .......... .......... 40% 139M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 38050K .......... .......... .......... .......... .......... 41% 90.3M 1s 2023-12-03T11:57:25.643 INFO:teuthology.orchestra.run.smithi116.stderr: 38100K .......... .......... .......... .......... .......... 41% 77.0M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38150K .......... .......... .......... .......... .......... 41% 236M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38200K .......... .......... .......... .......... .......... 41% 114M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38250K .......... .......... .......... .......... .......... 41% 115M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38300K .......... .......... .......... .......... .......... 41% 110M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38350K .......... .......... .......... .......... .......... 41% 117M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38400K .......... .......... .......... .......... .......... 41% 111M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38450K .......... .......... .......... .......... .......... 41% 94.0M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38500K .......... .......... .......... .......... .......... 41% 117M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38550K .......... .......... .......... .......... .......... 41% 112M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38600K .......... .......... .......... .......... .......... 41% 76.9M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38650K .......... .......... .......... .......... .......... 41% 95.4M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38700K .......... .......... .......... .......... .......... 41% 125M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38750K .......... .......... .......... .......... .......... 41% 103M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38800K .......... .......... .......... .......... .......... 41% 65.1M 1s 2023-12-03T11:57:25.644 INFO:teuthology.orchestra.run.smithi116.stderr: 38850K .......... .......... .......... .......... .......... 41% 83.1M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 38900K .......... .......... .......... .......... .......... 41% 110M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 38950K .......... .......... .......... .......... .......... 41% 119M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39000K .......... .......... .......... .......... .......... 42% 111M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39050K .......... .......... .......... .......... .......... 42% 116M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39100K .......... .......... .......... .......... .......... 42% 112M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39150K .......... .......... .......... .......... .......... 42% 122M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39200K .......... .......... .......... .......... .......... 42% 109M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39250K .......... .......... .......... .......... .......... 42% 89.7M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39300K .......... .......... .......... .......... .......... 42% 105M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39350K .......... .......... .......... .......... .......... 42% 118M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39400K .......... .......... .......... .......... .......... 42% 114M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39450K .......... .......... .......... .......... .......... 42% 106M 1s 2023-12-03T11:57:25.645 INFO:teuthology.orchestra.run.smithi116.stderr: 39500K .......... .......... .......... .......... .......... 42% 124M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39550K .......... .......... .......... .......... .......... 42% 112M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39600K .......... .......... .......... .......... .......... 42% 120M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39650K .......... .......... .......... .......... .......... 42% 88.2M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39700K .......... .......... .......... .......... .......... 42% 111M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39750K .......... .......... .......... .......... .......... 42% 124M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39800K .......... .......... .......... .......... .......... 42% 105M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39850K .......... .......... .......... .......... .......... 42% 111M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39900K .......... .......... .......... .......... .......... 43% 107M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 39950K .......... .......... .......... .......... .......... 43% 98.4M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40000K .......... .......... .......... .......... .......... 43% 95.0M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40050K .......... .......... .......... .......... .......... 43% 97.0M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40100K .......... .......... .......... .......... .......... 43% 132M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40150K .......... .......... .......... .......... .......... 43% 117M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40200K .......... .......... .......... .......... .......... 43% 78.6M 1s 2023-12-03T11:57:25.646 INFO:teuthology.orchestra.run.smithi116.stderr: 40250K .......... .......... .......... .......... .......... 43% 78.7M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40300K .......... .......... .......... .......... .......... 43% 115M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40350K .......... .......... .......... .......... .......... 43% 103M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40400K .......... .......... .......... .......... .......... 43% 131M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40450K .......... .......... .......... .......... .......... 43% 88.1M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40500K .......... .......... .......... .......... .......... 43% 116M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40550K .......... .......... .......... .......... .......... 43% 115M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40600K .......... .......... .......... .......... .......... 43% 115M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40650K .......... .......... .......... .......... .......... 43% 115M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40700K .......... .......... .......... .......... .......... 43% 113M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40750K .......... .......... .......... .......... .......... 43% 107M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40800K .......... .......... .......... .......... .......... 43% 122M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40850K .......... .......... .......... .......... .......... 44% 88.8M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40900K .......... .......... .......... .......... .......... 44% 116M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 40950K .......... .......... .......... .......... .......... 44% 110M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 41000K .......... .......... .......... .......... .......... 44% 122M 1s 2023-12-03T11:57:25.647 INFO:teuthology.orchestra.run.smithi116.stderr: 41050K .......... .......... .......... .......... .......... 44% 109M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41100K .......... .......... .......... .......... .......... 44% 119M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41150K .......... .......... .......... .......... .......... 44% 113M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41200K .......... .......... .......... .......... .......... 44% 113M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41250K .......... .......... .......... .......... .......... 44% 91.0M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41300K .......... .......... .......... .......... .......... 44% 112M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41350K .......... .......... .......... .......... .......... 44% 115M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41400K .......... .......... .......... .......... .......... 44% 109M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41450K .......... .......... .......... .......... .......... 44% 104M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41500K .......... .......... .......... .......... .......... 44% 127M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41550K .......... .......... .......... .......... .......... 44% 115M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41600K .......... .......... .......... .......... .......... 44% 110M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41650K .......... .......... .......... .......... .......... 44% 87.4M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41700K .......... .......... .......... .......... .......... 44% 121M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41750K .......... .......... .......... .......... .......... 44% 120M 1s 2023-12-03T11:57:25.648 INFO:teuthology.orchestra.run.smithi116.stderr: 41800K .......... .......... .......... .......... .......... 45% 114M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 41850K .......... .......... .......... .......... .......... 45% 111M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 41900K .......... .......... .......... .......... .......... 45% 114M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 41950K .......... .......... .......... .......... .......... 45% 115M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42000K .......... .......... .......... .......... .......... 45% 111M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42050K .......... .......... .......... .......... .......... 45% 89.2M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42100K .......... .......... .......... .......... .......... 45% 117M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42150K .......... .......... .......... .......... .......... 45% 114M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42200K .......... .......... .......... .......... .......... 45% 114M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42250K .......... .......... .......... .......... .......... 45% 108M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42300K .......... .......... .......... .......... .......... 45% 114M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42350K .......... .......... .......... .......... .......... 45% 117M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42400K .......... .......... .......... .......... .......... 45% 124M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42450K .......... .......... .......... .......... .......... 45% 88.5M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42500K .......... .......... .......... .......... .......... 45% 112M 1s 2023-12-03T11:57:25.649 INFO:teuthology.orchestra.run.smithi116.stderr: 42550K .......... .......... .......... .......... .......... 45% 91.2M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42600K .......... .......... .......... .......... .......... 45% 96.0M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42650K .......... .......... .......... .......... .......... 45% 95.0M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42700K .......... .......... .......... .......... .......... 46% 111M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42750K .......... .......... .......... .......... .......... 46% 32.3M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42800K .......... .......... .......... .......... .......... 46% 119M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42850K .......... .......... .......... .......... .......... 46% 79.7M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42900K .......... .......... .......... .......... .......... 46% 101M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 42950K .......... .......... .......... .......... .......... 46% 131M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43000K .......... .......... .......... .......... .......... 46% 110M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43050K .......... .......... .......... .......... .......... 46% 117M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43100K .......... .......... .......... .......... .......... 46% 104M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43150K .......... .......... .......... .......... .......... 46% 115M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43200K .......... .......... .......... .......... .......... 46% 96.8M 1s 2023-12-03T11:57:25.650 INFO:teuthology.orchestra.run.smithi116.stderr: 43250K .......... .......... .......... .......... .......... 46% 98.1M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43300K .......... .......... .......... .......... .......... 46% 111M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43350K .......... .......... .......... .......... .......... 46% 117M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43400K .......... .......... .......... .......... .......... 46% 104M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43450K .......... .......... .......... .......... .......... 46% 123M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43500K .......... .......... .......... .......... .......... 46% 112M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43550K .......... .......... .......... .......... .......... 46% 111M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43600K .......... .......... .......... .......... .......... 46% 117M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43650K .......... .......... .......... .......... .......... 47% 94.8M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43700K .......... .......... .......... .......... .......... 47% 107M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43750K .......... .......... .......... .......... .......... 47% 105M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43800K .......... .......... .......... .......... .......... 47% 115M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43850K .......... .......... .......... .......... .......... 47% 113M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43900K .......... .......... .......... .......... .......... 47% 113M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 43950K .......... .......... .......... .......... .......... 47% 105M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 44000K .......... .......... .......... .......... .......... 47% 113M 1s 2023-12-03T11:57:25.651 INFO:teuthology.orchestra.run.smithi116.stderr: 44050K .......... .......... .......... .......... .......... 47% 73.1M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44100K .......... .......... .......... .......... .......... 47% 114M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44150K .......... .......... .......... .......... .......... 47% 50.2M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44200K .......... .......... .......... .......... .......... 47% 65.1M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44250K .......... .......... .......... .......... .......... 47% 104M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44300K .......... .......... .......... .......... .......... 47% 116M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44350K .......... .......... .......... .......... .......... 47% 117M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44400K .......... .......... .......... .......... .......... 47% 109M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44450K .......... .......... .......... .......... .......... 47% 92.7M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44500K .......... .......... .......... .......... .......... 47% 109M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44550K .......... .......... .......... .......... .......... 48% 119M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44600K .......... .......... .......... .......... .......... 48% 111M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44650K .......... .......... .......... .......... .......... 48% 113M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44700K .......... .......... .......... .......... .......... 48% 113M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44750K .......... .......... .......... .......... .......... 48% 117M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44800K .......... .......... .......... .......... .......... 48% 105M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44850K .......... .......... .......... .......... .......... 48% 91.4M 1s 2023-12-03T11:57:25.652 INFO:teuthology.orchestra.run.smithi116.stderr: 44900K .......... .......... .......... .......... .......... 48% 111M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 44950K .......... .......... .......... .......... .......... 48% 111M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45000K .......... .......... .......... .......... .......... 48% 129M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45050K .......... .......... .......... .......... .......... 48% 105M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45100K .......... .......... .......... .......... .......... 48% 116M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45150K .......... .......... .......... .......... .......... 48% 115M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45200K .......... .......... .......... .......... .......... 48% 112M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45250K .......... .......... .......... .......... .......... 48% 96.9M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45300K .......... .......... .......... .......... .......... 48% 111M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45350K .......... .......... .......... .......... .......... 48% 121M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45400K .......... .......... .......... .......... .......... 48% 97.9M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45450K .......... .......... .......... .......... .......... 48% 110M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45500K .......... .......... .......... .......... .......... 49% 110M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45550K .......... .......... .......... .......... .......... 49% 122M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45600K .......... .......... .......... .......... .......... 49% 71.0M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45650K .......... .......... .......... .......... .......... 49% 147M 1s 2023-12-03T11:57:25.653 INFO:teuthology.orchestra.run.smithi116.stderr: 45700K .......... .......... .......... .......... .......... 49% 122M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 45750K .......... .......... .......... .......... .......... 49% 107M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 45800K .......... .......... .......... .......... .......... 49% 120M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 45850K .......... .......... .......... .......... .......... 49% 112M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 45900K .......... .......... .......... .......... .......... 49% 112M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 45950K .......... .......... .......... .......... .......... 49% 116M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46000K .......... .......... .......... .......... .......... 49% 120M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46050K .......... .......... .......... .......... .......... 49% 86.8M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46100K .......... .......... .......... .......... .......... 49% 113M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46150K .......... .......... .......... .......... .......... 49% 114M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46200K .......... .......... .......... .......... .......... 49% 112M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46250K .......... .......... .......... .......... .......... 49% 110M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46300K .......... .......... .......... .......... .......... 49% 122M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46350K .......... .......... .......... .......... .......... 49% 109M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46400K .......... .......... .......... .......... .......... 49% 118M 1s 2023-12-03T11:57:25.654 INFO:teuthology.orchestra.run.smithi116.stderr: 46450K .......... .......... .......... .......... .......... 50% 90.2M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46500K .......... .......... .......... .......... .......... 50% 90.7M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46550K .......... .......... .......... .......... .......... 50% 92.5M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46600K .......... .......... .......... .......... .......... 50% 110M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46650K .......... .......... .......... .......... .......... 50% 112M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46700K .......... .......... .......... .......... .......... 50% 29.4M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46750K .......... .......... .......... .......... .......... 50% 158M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46800K .......... .......... .......... .......... .......... 50% 99.6M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46850K .......... .......... .......... .......... .......... 50% 93.2M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46900K .......... .......... .......... .......... .......... 50% 113M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 46950K .......... .......... .......... .......... .......... 50% 117M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47000K .......... .......... .......... .......... .......... 50% 112M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47050K .......... .......... .......... .......... .......... 50% 92.5M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47100K .......... .......... .......... .......... .......... 50% 114M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47150K .......... .......... .......... .......... .......... 50% 121M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47200K .......... .......... .......... .......... .......... 50% 97.3M 1s 2023-12-03T11:57:25.655 INFO:teuthology.orchestra.run.smithi116.stderr: 47250K .......... .......... .......... .......... .......... 50% 98.1M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47300K .......... .......... .......... .......... .......... 50% 115M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47350K .......... .......... .......... .......... .......... 51% 115M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47400K .......... .......... .......... .......... .......... 51% 115M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47450K .......... .......... .......... .......... .......... 51% 104M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47500K .......... .......... .......... .......... .......... 51% 113M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47550K .......... .......... .......... .......... .......... 51% 113M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47600K .......... .......... .......... .......... .......... 51% 119M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47650K .......... .......... .......... .......... .......... 51% 88.4M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47700K .......... .......... .......... .......... .......... 51% 107M 1s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47750K .......... .......... .......... .......... .......... 51% 114M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47800K .......... .......... .......... .......... .......... 51% 117M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47850K .......... .......... .......... .......... .......... 51% 112M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47900K .......... .......... .......... .......... .......... 51% 113M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 47950K .......... .......... .......... .......... .......... 51% 94.4M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 48000K .......... .......... .......... .......... .......... 51% 106M 0s 2023-12-03T11:57:25.656 INFO:teuthology.orchestra.run.smithi116.stderr: 48050K .......... .......... .......... .......... .......... 51% 26.9M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48100K .......... .......... .......... .......... .......... 51% 126M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48150K .......... .......... .......... .......... .......... 51% 60.1M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48200K .......... .......... .......... .......... .......... 51% 118M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48250K .......... .......... .......... .......... .......... 51% 108M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48300K .......... .......... .......... .......... .......... 52% 117M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48350K .......... .......... .......... .......... .......... 52% 115M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48400K .......... .......... .......... .......... .......... 52% 114M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48450K .......... .......... .......... .......... .......... 52% 89.6M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48500K .......... .......... .......... .......... .......... 52% 122M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48550K .......... .......... .......... .......... .......... 52% 115M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48600K .......... .......... .......... .......... .......... 52% 115M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48650K .......... .......... .......... .......... .......... 52% 109M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48700K .......... .......... .......... .......... .......... 52% 109M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48750K .......... .......... .......... .......... .......... 52% 113M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48800K .......... .......... .......... .......... .......... 52% 127M 0s 2023-12-03T11:57:25.657 INFO:teuthology.orchestra.run.smithi116.stderr: 48850K .......... .......... .......... .......... .......... 52% 78.3M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 48900K .......... .......... .......... .......... .......... 52% 142M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 48950K .......... .......... .......... .......... .......... 52% 110M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49000K .......... .......... .......... .......... .......... 52% 118M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49050K .......... .......... .......... .......... .......... 52% 108M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49100K .......... .......... .......... .......... .......... 52% 126M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49150K .......... .......... .......... .......... .......... 52% 109M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49200K .......... .......... .......... .......... .......... 53% 122M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49250K .......... .......... .......... .......... .......... 53% 73.7M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49300K .......... .......... .......... .......... .......... 53% 154M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49350K .......... .......... .......... .......... .......... 53% 118M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49400K .......... .......... .......... .......... .......... 53% 93.4M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49450K .......... .......... .......... .......... .......... 53% 121M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49500K .......... .......... .......... .......... .......... 53% 111M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49550K .......... .......... .......... .......... .......... 53% 111M 0s 2023-12-03T11:57:25.658 INFO:teuthology.orchestra.run.smithi116.stderr: 49600K .......... .......... .......... .......... .......... 53% 120M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49650K .......... .......... .......... .......... .......... 53% 85.8M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49700K .......... .......... .......... .......... .......... 53% 121M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49750K .......... .......... .......... .......... .......... 53% 111M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49800K .......... .......... .......... .......... .......... 53% 116M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49850K .......... .......... .......... .......... .......... 53% 114M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49900K .......... .......... .......... .......... .......... 53% 115M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 49950K .......... .......... .......... .......... .......... 53% 97.8M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50000K .......... .......... .......... .......... .......... 53% 112M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50050K .......... .......... .......... .......... .......... 53% 89.1M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50100K .......... .......... .......... .......... .......... 53% 111M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50150K .......... .......... .......... .......... .......... 54% 124M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50200K .......... .......... .......... .......... .......... 54% 113M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50250K .......... .......... .......... .......... .......... 54% 110M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50300K .......... .......... .......... .......... .......... 54% 115M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50350K .......... .......... .......... .......... .......... 54% 120M 0s 2023-12-03T11:57:25.659 INFO:teuthology.orchestra.run.smithi116.stderr: 50400K .......... .......... .......... .......... .......... 54% 118M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50450K .......... .......... .......... .......... .......... 54% 72.2M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50500K .......... .......... .......... .......... .......... 54% 102M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50550K .......... .......... .......... .......... .......... 54% 110M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50600K .......... .......... .......... .......... .......... 54% 118M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50650K .......... .......... .......... .......... .......... 54% 35.6M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50700K .......... .......... .......... .......... .......... 54% 68.1M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50750K .......... .......... .......... .......... .......... 54% 99.1M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50800K .......... .......... .......... .......... .......... 54% 117M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50850K .......... .......... .......... .......... .......... 54% 91.9M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50900K .......... .......... .......... .......... .......... 54% 114M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 50950K .......... .......... .......... .......... .......... 54% 116M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 51000K .......... .......... .......... .......... .......... 54% 108M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 51050K .......... .......... .......... .......... .......... 55% 102M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 51100K .......... .......... .......... .......... .......... 55% 116M 0s 2023-12-03T11:57:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 51150K .......... .......... .......... .......... .......... 55% 112M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51200K .......... .......... .......... .......... .......... 55% 105M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51250K .......... .......... .......... .......... .......... 55% 87.8M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51300K .......... .......... .......... .......... .......... 55% 114M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51350K .......... .......... .......... .......... .......... 55% 118M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51400K .......... .......... .......... .......... .......... 55% 117M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51450K .......... .......... .......... .......... .......... 55% 110M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51500K .......... .......... .......... .......... .......... 55% 112M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51550K .......... .......... .......... .......... .......... 55% 120M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51600K .......... .......... .......... .......... .......... 55% 107M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51650K .......... .......... .......... .......... .......... 55% 92.9M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51700K .......... .......... .......... .......... .......... 55% 98.1M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51750K .......... .......... .......... .......... .......... 55% 109M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51800K .......... .......... .......... .......... .......... 55% 116M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51850K .......... .......... .......... .......... .......... 55% 115M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51900K .......... .......... .......... .......... .......... 55% 94.7M 0s 2023-12-03T11:57:25.661 INFO:teuthology.orchestra.run.smithi116.stderr: 51950K .......... .......... .......... .......... .......... 55% 51.5M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52000K .......... .......... .......... .......... .......... 56% 119M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52050K .......... .......... .......... .......... .......... 56% 90.1M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52100K .......... .......... .......... .......... .......... 56% 107M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52150K .......... .......... .......... .......... .......... 56% 114M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52200K .......... .......... .......... .......... .......... 56% 109M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52250K .......... .......... .......... .......... .......... 56% 108M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52300K .......... .......... .......... .......... .......... 56% 42.8M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52350K .......... .......... .......... .......... .......... 56% 118M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52400K .......... .......... .......... .......... .......... 56% 101M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52450K .......... .......... .......... .......... .......... 56% 89.5M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52500K .......... .......... .......... .......... .......... 56% 112M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52550K .......... .......... .......... .......... .......... 56% 109M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52600K .......... .......... .......... .......... .......... 56% 104M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52650K .......... .......... .......... .......... .......... 56% 114M 0s 2023-12-03T11:57:25.662 INFO:teuthology.orchestra.run.smithi116.stderr: 52700K .......... .......... .......... .......... .......... 56% 113M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 52750K .......... .......... .......... .......... .......... 56% 106M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 52800K .......... .......... .......... .......... .......... 56% 115M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 52850K .......... .......... .......... .......... .......... 56% 88.7M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 52900K .......... .......... .......... .......... .......... 56% 105M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 52950K .......... .......... .......... .......... .......... 57% 113M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53000K .......... .......... .......... .......... .......... 57% 117M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53050K .......... .......... .......... .......... .......... 57% 109M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53100K .......... .......... .......... .......... .......... 57% 118M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53150K .......... .......... .......... .......... .......... 57% 112M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53200K .......... .......... .......... .......... .......... 57% 110M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53250K .......... .......... .......... .......... .......... 57% 89.1M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53300K .......... .......... .......... .......... .......... 57% 103M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53350K .......... .......... .......... .......... .......... 57% 97.7M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53400K .......... .......... .......... .......... .......... 57% 105M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53450K .......... .......... .......... .......... .......... 57% 113M 0s 2023-12-03T11:57:25.663 INFO:teuthology.orchestra.run.smithi116.stderr: 53500K .......... .......... .......... .......... .......... 57% 115M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53550K .......... .......... .......... .......... .......... 57% 113M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53600K .......... .......... .......... .......... .......... 57% 111M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53650K .......... .......... .......... .......... .......... 57% 91.4M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53700K .......... .......... .......... .......... .......... 57% 116M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53750K .......... .......... .......... .......... .......... 57% 104M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53800K .......... .......... .......... .......... .......... 57% 123M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53850K .......... .......... .......... .......... .......... 58% 110M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53900K .......... .......... .......... .......... .......... 58% 115M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 53950K .......... .......... .......... .......... .......... 58% 112M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54000K .......... .......... .......... .......... .......... 58% 113M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54050K .......... .......... .......... .......... .......... 58% 92.9M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54100K .......... .......... .......... .......... .......... 58% 106M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54150K .......... .......... .......... .......... .......... 58% 119M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54200K .......... .......... .......... .......... .......... 58% 107M 0s 2023-12-03T11:57:25.664 INFO:teuthology.orchestra.run.smithi116.stderr: 54250K .......... .......... .......... .......... .......... 58% 112M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54300K .......... .......... .......... .......... .......... 58% 110M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54350K .......... .......... .......... .......... .......... 58% 119M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54400K .......... .......... .......... .......... .......... 58% 90.9M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54450K .......... .......... .......... .......... .......... 58% 84.3M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54500K .......... .......... .......... .......... .......... 58% 91.1M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54550K .......... .......... .......... .......... .......... 58% 42.3M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54600K .......... .......... .......... .......... .......... 58% 103M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54650K .......... .......... .......... .......... .......... 58% 64.9M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54700K .......... .......... .......... .......... .......... 58% 117M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54750K .......... .......... .......... .......... .......... 58% 107M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54800K .......... .......... .......... .......... .......... 59% 112M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54850K .......... .......... .......... .......... .......... 59% 88.2M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54900K .......... .......... .......... .......... .......... 59% 117M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 54950K .......... .......... .......... .......... .......... 59% 116M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 55000K .......... .......... .......... .......... .......... 59% 103M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 55050K .......... .......... .......... .......... .......... 59% 111M 0s 2023-12-03T11:57:25.665 INFO:teuthology.orchestra.run.smithi116.stderr: 55100K .......... .......... .......... .......... .......... 59% 116M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55150K .......... .......... .......... .......... .......... 59% 112M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55200K .......... .......... .......... .......... .......... 59% 116M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55250K .......... .......... .......... .......... .......... 59% 87.4M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55300K .......... .......... .......... .......... .......... 59% 120M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55350K .......... .......... .......... .......... .......... 59% 115M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55400K .......... .......... .......... .......... .......... 59% 116M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55450K .......... .......... .......... .......... .......... 59% 112M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55500K .......... .......... .......... .......... .......... 59% 114M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55550K .......... .......... .......... .......... .......... 59% 121M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55600K .......... .......... .......... .......... .......... 59% 112M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55650K .......... .......... .......... .......... .......... 59% 75.9M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55700K .......... .......... .......... .......... .......... 60% 112M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55750K .......... .......... .......... .......... .......... 60% 117M 0s 2023-12-03T11:57:25.666 INFO:teuthology.orchestra.run.smithi116.stderr: 55800K .......... .......... .......... .......... .......... 60% 116M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 55850K .......... .......... .......... .......... .......... 60% 104M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 55900K .......... .......... .......... .......... .......... 60% 35.4M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 55950K .......... .......... .......... .......... .......... 60% 115M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56000K .......... .......... .......... .......... .......... 60% 116M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56050K .......... .......... .......... .......... .......... 60% 87.5M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56100K .......... .......... .......... .......... .......... 60% 117M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56150K .......... .......... .......... .......... .......... 60% 113M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56200K .......... .......... .......... .......... .......... 60% 110M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56250K .......... .......... .......... .......... .......... 60% 97.3M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56300K .......... .......... .......... .......... .......... 60% 62.9M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56350K .......... .......... .......... .......... .......... 60% 18.3M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56400K .......... .......... .......... .......... .......... 60% 111M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56450K .......... .......... .......... .......... .......... 60% 91.7M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56500K .......... .......... .......... .......... .......... 60% 115M 0s 2023-12-03T11:57:25.667 INFO:teuthology.orchestra.run.smithi116.stderr: 56550K .......... .......... .......... .......... .......... 60% 118M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56600K .......... .......... .......... .......... .......... 60% 107M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56650K .......... .......... .......... .......... .......... 61% 110M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56700K .......... .......... .......... .......... .......... 61% 114M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56750K .......... .......... .......... .......... .......... 61% 118M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56800K .......... .......... .......... .......... .......... 61% 94.5M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56850K .......... .......... .......... .......... .......... 61% 103M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56900K .......... .......... .......... .......... .......... 61% 115M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 56950K .......... .......... .......... .......... .......... 61% 112M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57000K .......... .......... .......... .......... .......... 61% 119M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57050K .......... .......... .......... .......... .......... 61% 105M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57100K .......... .......... .......... .......... .......... 61% 120M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57150K .......... .......... .......... .......... .......... 61% 120M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57200K .......... .......... .......... .......... .......... 61% 109M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57250K .......... .......... .......... .......... .......... 61% 89.9M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57300K .......... .......... .......... .......... .......... 61% 113M 0s 2023-12-03T11:57:25.668 INFO:teuthology.orchestra.run.smithi116.stderr: 57350K .......... .......... .......... .......... .......... 61% 103M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57400K .......... .......... .......... .......... .......... 61% 110M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57450K .......... .......... .......... .......... .......... 61% 114M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57500K .......... .......... .......... .......... .......... 61% 113M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57550K .......... .......... .......... .......... .......... 62% 109M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57600K .......... .......... .......... .......... .......... 62% 113M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57650K .......... .......... .......... .......... .......... 62% 86.2M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57700K .......... .......... .......... .......... .......... 62% 123M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57750K .......... .......... .......... .......... .......... 62% 116M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57800K .......... .......... .......... .......... .......... 62% 114M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57850K .......... .......... .......... .......... .......... 62% 107M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57900K .......... .......... .......... .......... .......... 62% 120M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 57950K .......... .......... .......... .......... .......... 62% 108M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 58000K .......... .......... .......... .......... .......... 62% 105M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 58050K .......... .......... .......... .......... .......... 62% 88.2M 0s 2023-12-03T11:57:25.669 INFO:teuthology.orchestra.run.smithi116.stderr: 58100K .......... .......... .......... .......... .......... 62% 116M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58150K .......... .......... .......... .......... .......... 62% 117M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58200K .......... .......... .......... .......... .......... 62% 113M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58250K .......... .......... .......... .......... .......... 62% 106M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58300K .......... .......... .......... .......... .......... 62% 114M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58350K .......... .......... .......... .......... .......... 62% 86.6M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58400K .......... .......... .......... .......... .......... 62% 116M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58450K .......... .......... .......... .......... .......... 62% 72.9M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58500K .......... .......... .......... .......... .......... 63% 39.4M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58550K .......... .......... .......... .......... .......... 63% 104M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58600K .......... .......... .......... .......... .......... 63% 65.2M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58650K .......... .......... .......... .......... .......... 63% 107M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58700K .......... .......... .......... .......... .......... 63% 118M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58750K .......... .......... .......... .......... .......... 63% 119M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58800K .......... .......... .......... .......... .......... 63% 112M 0s 2023-12-03T11:57:25.670 INFO:teuthology.orchestra.run.smithi116.stderr: 58850K .......... .......... .......... .......... .......... 63% 89.2M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 58900K .......... .......... .......... .......... .......... 63% 105M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 58950K .......... .......... .......... .......... .......... 63% 116M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59000K .......... .......... .......... .......... .......... 63% 92.5M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59050K .......... .......... .......... .......... .......... 63% 115M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59100K .......... .......... .......... .......... .......... 63% 109M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59150K .......... .......... .......... .......... .......... 63% 107M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59200K .......... .......... .......... .......... .......... 63% 113M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59250K .......... .......... .......... .......... .......... 63% 85.0M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59300K .......... .......... .......... .......... .......... 63% 118M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59350K .......... .......... .......... .......... .......... 63% 113M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59400K .......... .......... .......... .......... .......... 63% 115M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59450K .......... .......... .......... .......... .......... 64% 109M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59500K .......... .......... .......... .......... .......... 64% 120M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59550K .......... .......... .......... .......... .......... 64% 104M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59600K .......... .......... .......... .......... .......... 64% 97.1M 0s 2023-12-03T11:57:25.671 INFO:teuthology.orchestra.run.smithi116.stderr: 59650K .......... .......... .......... .......... .......... 64% 85.8M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59700K .......... .......... .......... .......... .......... 64% 108M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59750K .......... .......... .......... .......... .......... 64% 120M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59800K .......... .......... .......... .......... .......... 64% 110M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59850K .......... .......... .......... .......... .......... 64% 48.8M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59900K .......... .......... .......... .......... .......... 64% 121M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 59950K .......... .......... .......... .......... .......... 64% 104M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60000K .......... .......... .......... .......... .......... 64% 118M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60050K .......... .......... .......... .......... .......... 64% 87.6M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60100K .......... .......... .......... .......... .......... 64% 104M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60150K .......... .......... .......... .......... .......... 64% 115M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60200K .......... .......... .......... .......... .......... 64% 111M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60250K .......... .......... .......... .......... .......... 64% 34.8M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60300K .......... .......... .......... .......... .......... 64% 16.5M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60350K .......... .......... .......... .......... .......... 65% 163M 0s 2023-12-03T11:57:25.672 INFO:teuthology.orchestra.run.smithi116.stderr: 60400K .......... .......... .......... .......... .......... 65% 113M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60450K .......... .......... .......... .......... .......... 65% 89.3M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60500K .......... .......... .......... .......... .......... 65% 97.8M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60550K .......... .......... .......... .......... .......... 65% 133M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60600K .......... .......... .......... .......... .......... 65% 115M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60650K .......... .......... .......... .......... .......... 65% 110M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60700K .......... .......... .......... .......... .......... 65% 115M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60750K .......... .......... .......... .......... .......... 65% 115M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60800K .......... .......... .......... .......... .......... 65% 118M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60850K .......... .......... .......... .......... .......... 65% 85.4M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60900K .......... .......... .......... .......... .......... 65% 115M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 60950K .......... .......... .......... .......... .......... 65% 114M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 61000K .......... .......... .......... .......... .......... 65% 118M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 61050K .......... .......... .......... .......... .......... 65% 110M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 61100K .......... .......... .......... .......... .......... 65% 112M 0s 2023-12-03T11:57:25.673 INFO:teuthology.orchestra.run.smithi116.stderr: 61150K .......... .......... .......... .......... .......... 65% 121M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61200K .......... .......... .......... .......... .......... 65% 113M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61250K .......... .......... .......... .......... .......... 65% 89.8M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61300K .......... .......... .......... .......... .......... 66% 99.3M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61350K .......... .......... .......... .......... .......... 66% 118M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61400K .......... .......... .......... .......... .......... 66% 110M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61450K .......... .......... .......... .......... .......... 66% 113M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61500K .......... .......... .......... .......... .......... 66% 118M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61550K .......... .......... .......... .......... .......... 66% 115M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61600K .......... .......... .......... .......... .......... 66% 119M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61650K .......... .......... .......... .......... .......... 66% 87.3M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61700K .......... .......... .......... .......... .......... 66% 116M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61750K .......... .......... .......... .......... .......... 66% 111M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61800K .......... .......... .......... .......... .......... 66% 120M 0s 2023-12-03T11:57:25.674 INFO:teuthology.orchestra.run.smithi116.stderr: 61850K .......... .......... .......... .......... .......... 66% 110M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 61900K .......... .......... .......... .......... .......... 66% 116M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 61950K .......... .......... .......... .......... .......... 66% 106M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62000K .......... .......... .......... .......... .......... 66% 108M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62050K .......... .......... .......... .......... .......... 66% 79.3M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62100K .......... .......... .......... .......... .......... 66% 120M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62150K .......... .......... .......... .......... .......... 66% 96.0M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62200K .......... .......... .......... .......... .......... 67% 114M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62250K .......... .......... .......... .......... .......... 67% 88.1M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62300K .......... .......... .......... .......... .......... 67% 120M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62350K .......... .......... .......... .......... .......... 67% 98.6M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62400K .......... .......... .......... .......... .......... 67% 84.2M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62450K .......... .......... .......... .......... .......... 67% 36.2M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62500K .......... .......... .......... .......... .......... 67% 62.4M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62550K .......... .......... .......... .......... .......... 67% 112M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62600K .......... .......... .......... .......... .......... 67% 121M 0s 2023-12-03T11:57:25.675 INFO:teuthology.orchestra.run.smithi116.stderr: 62650K .......... .......... .......... .......... .......... 67% 108M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62700K .......... .......... .......... .......... .......... 67% 114M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62750K .......... .......... .......... .......... .......... 67% 120M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62800K .......... .......... .......... .......... .......... 67% 115M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62850K .......... .......... .......... .......... .......... 67% 89.1M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62900K .......... .......... .......... .......... .......... 67% 103M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 62950K .......... .......... .......... .......... .......... 67% 99.4M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63000K .......... .......... .......... .......... .......... 67% 110M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63050K .......... .......... .......... .......... .......... 67% 113M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63100K .......... .......... .......... .......... .......... 67% 120M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63150K .......... .......... .......... .......... .......... 68% 119M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63200K .......... .......... .......... .......... .......... 68% 113M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63250K .......... .......... .......... .......... .......... 68% 87.2M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63300K .......... .......... .......... .......... .......... 68% 107M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63350K .......... .......... .......... .......... .......... 68% 121M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63400K .......... .......... .......... .......... .......... 68% 112M 0s 2023-12-03T11:57:25.676 INFO:teuthology.orchestra.run.smithi116.stderr: 63450K .......... .......... .......... .......... .......... 68% 111M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63500K .......... .......... .......... .......... .......... 68% 113M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63550K .......... .......... .......... .......... .......... 68% 98.3M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63600K .......... .......... .......... .......... .......... 68% 101M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63650K .......... .......... .......... .......... .......... 68% 86.9M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63700K .......... .......... .......... .......... .......... 68% 110M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63750K .......... .......... .......... .......... .......... 68% 119M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63800K .......... .......... .......... .......... .......... 68% 88.6M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63850K .......... .......... .......... .......... .......... 68% 118M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63900K .......... .......... .......... .......... .......... 68% 116M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 63950K .......... .......... .......... .......... .......... 68% 113M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64000K .......... .......... .......... .......... .......... 68% 109M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64050K .......... .......... .......... .......... .......... 68% 88.4M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64100K .......... .......... .......... .......... .......... 69% 108M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64150K .......... .......... .......... .......... .......... 69% 34.2M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64200K .......... .......... .......... .......... .......... 69% 119M 0s 2023-12-03T11:57:25.677 INFO:teuthology.orchestra.run.smithi116.stderr: 64250K .......... .......... .......... .......... .......... 69% 94.3M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64300K .......... .......... .......... .......... .......... 69% 117M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64350K .......... .......... .......... .......... .......... 69% 128M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64400K .......... .......... .......... .......... .......... 69% 110M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64450K .......... .......... .......... .......... .......... 69% 85.1M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64500K .......... .......... .......... .......... .......... 69% 31.9M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64550K .......... .......... .......... .......... .......... 69% 104M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64600K .......... .......... .......... .......... .......... 69% 114M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64650K .......... .......... .......... .......... .......... 69% 110M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64700K .......... .......... .......... .......... .......... 69% 117M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64750K .......... .......... .......... .......... .......... 69% 120M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64800K .......... .......... .......... .......... .......... 69% 116M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64850K .......... .......... .......... .......... .......... 69% 88.4M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64900K .......... .......... .......... .......... .......... 69% 113M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 64950K .......... .......... .......... .......... .......... 69% 110M 0s 2023-12-03T11:57:25.678 INFO:teuthology.orchestra.run.smithi116.stderr: 65000K .......... .......... .......... .......... .......... 70% 105M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65050K .......... .......... .......... .......... .......... 70% 117M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65100K .......... .......... .......... .......... .......... 70% 113M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65150K .......... .......... .......... .......... .......... 70% 118M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65200K .......... .......... .......... .......... .......... 70% 115M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65250K .......... .......... .......... .......... .......... 70% 78.1M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65300K .......... .......... .......... .......... .......... 70% 107M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65350K .......... .......... .......... .......... .......... 70% 118M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65400K .......... .......... .......... .......... .......... 70% 114M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65450K .......... .......... .......... .......... .......... 70% 110M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65500K .......... .......... .......... .......... .......... 70% 107M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65550K .......... .......... .......... .......... .......... 70% 106M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65600K .......... .......... .......... .......... .......... 70% 121M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65650K .......... .......... .......... .......... .......... 70% 90.7M 0s 2023-12-03T11:57:25.679 INFO:teuthology.orchestra.run.smithi116.stderr: 65700K .......... .......... .......... .......... .......... 70% 115M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 65750K .......... .......... .......... .......... .......... 70% 109M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 65800K .......... .......... .......... .......... .......... 70% 97.1M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 65850K .......... .......... .......... .......... .......... 70% 121M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 65900K .......... .......... .......... .......... .......... 70% 94.7M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 65950K .......... .......... .......... .......... .......... 71% 148M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66000K .......... .......... .......... .......... .......... 71% 111M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66050K .......... .......... .......... .......... .......... 71% 61.7M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66100K .......... .......... .......... .......... .......... 71% 13.3M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66150K .......... .......... .......... .......... .......... 71% 149M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66200K .......... .......... .......... .......... .......... 71% 92.2M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66250K .......... .......... .......... .......... .......... 71% 111M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66300K .......... .......... .......... .......... .......... 71% 121M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66350K .......... .......... .......... .......... .......... 71% 74.7M 0s 2023-12-03T11:57:25.680 INFO:teuthology.orchestra.run.smithi116.stderr: 66400K .......... .......... .......... .......... .......... 71% 40.4M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66450K .......... .......... .......... .......... .......... 71% 54.1M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66500K .......... .......... .......... .......... .......... 71% 115M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66550K .......... .......... .......... .......... .......... 71% 113M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66600K .......... .......... .......... .......... .......... 71% 116M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66650K .......... .......... .......... .......... .......... 71% 102M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66700K .......... .......... .......... .......... .......... 71% 121M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66750K .......... .......... .......... .......... .......... 71% 111M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66800K .......... .......... .......... .......... .......... 71% 113M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66850K .......... .......... .......... .......... .......... 72% 92.8M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66900K .......... .......... .......... .......... .......... 72% 93.9M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 66950K .......... .......... .......... .......... .......... 72% 96.8M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 67000K .......... .......... .......... .......... .......... 72% 122M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 67050K .......... .......... .......... .......... .......... 72% 109M 0s 2023-12-03T11:57:25.681 INFO:teuthology.orchestra.run.smithi116.stderr: 67100K .......... .......... .......... .......... .......... 72% 118M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67150K .......... .......... .......... .......... .......... 72% 102M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67200K .......... .......... .......... .......... .......... 72% 127M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67250K .......... .......... .......... .......... .......... 72% 82.2M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67300K .......... .......... .......... .......... .......... 72% 128M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67350K .......... .......... .......... .......... .......... 72% 106M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67400K .......... .......... .......... .......... .......... 72% 110M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67450K .......... .......... .......... .......... .......... 72% 112M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67500K .......... .......... .......... .......... .......... 72% 97.4M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67550K .......... .......... .......... .......... .......... 72% 105M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67600K .......... .......... .......... .......... .......... 72% 97.8M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67650K .......... .......... .......... .......... .......... 72% 95.6M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67700K .......... .......... .......... .......... .......... 72% 114M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67750K .......... .......... .......... .......... .......... 72% 97.1M 0s 2023-12-03T11:57:25.682 INFO:teuthology.orchestra.run.smithi116.stderr: 67800K .......... .......... .......... .......... .......... 73% 111M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 67850K .......... .......... .......... .......... .......... 73% 112M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 67900K .......... .......... .......... .......... .......... 73% 121M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 67950K .......... .......... .......... .......... .......... 73% 102M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68000K .......... .......... .......... .......... .......... 73% 108M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68050K .......... .......... .......... .......... .......... 73% 85.5M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68100K .......... .......... .......... .......... .......... 73% 21.2M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68150K .......... .......... .......... .......... .......... 73% 124M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68200K .......... .......... .......... .......... .......... 73% 116M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68250K .......... .......... .......... .......... .......... 73% 112M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68300K .......... .......... .......... .......... .......... 73% 108M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68350K .......... .......... .......... .......... .......... 73% 107M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68400K .......... .......... .......... .......... .......... 73% 118M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68450K .......... .......... .......... .......... .......... 73% 17.1M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68500K .......... .......... .......... .......... .......... 73% 115M 0s 2023-12-03T11:57:25.683 INFO:teuthology.orchestra.run.smithi116.stderr: 68550K .......... .......... .......... .......... .......... 73% 103M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68600K .......... .......... .......... .......... .......... 73% 121M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68650K .......... .......... .......... .......... .......... 73% 104M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68700K .......... .......... .......... .......... .......... 74% 111M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68750K .......... .......... .......... .......... .......... 74% 113M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68800K .......... .......... .......... .......... .......... 74% 119M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68850K .......... .......... .......... .......... .......... 74% 89.4M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68900K .......... .......... .......... .......... .......... 74% 104M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 68950K .......... .......... .......... .......... .......... 74% 118M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69000K .......... .......... .......... .......... .......... 74% 117M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69050K .......... .......... .......... .......... .......... 74% 114M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69100K .......... .......... .......... .......... .......... 74% 108M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69150K .......... .......... .......... .......... .......... 74% 109M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69200K .......... .......... .......... .......... .......... 74% 98.8M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69250K .......... .......... .......... .......... .......... 74% 87.3M 0s 2023-12-03T11:57:25.684 INFO:teuthology.orchestra.run.smithi116.stderr: 69300K .......... .......... .......... .......... .......... 74% 110M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69350K .......... .......... .......... .......... .......... 74% 119M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69400K .......... .......... .......... .......... .......... 74% 104M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69450K .......... .......... .......... .......... .......... 74% 106M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69500K .......... .......... .......... .......... .......... 74% 117M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69550K .......... .......... .......... .......... .......... 74% 113M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69600K .......... .......... .......... .......... .......... 74% 102M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69650K .......... .......... .......... .......... .......... 75% 87.6M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69700K .......... .......... .......... .......... .......... 75% 122M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69750K .......... .......... .......... .......... .......... 75% 116M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69800K .......... .......... .......... .......... .......... 75% 118M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69850K .......... .......... .......... .......... .......... 75% 102M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69900K .......... .......... .......... .......... .......... 75% 116M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 69950K .......... .......... .......... .......... .......... 75% 87.5M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 70000K .......... .......... .......... .......... .......... 75% 106M 0s 2023-12-03T11:57:25.685 INFO:teuthology.orchestra.run.smithi116.stderr: 70050K .......... .......... .......... .......... .......... 75% 19.5M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70100K .......... .......... .......... .......... .......... 75% 130M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70150K .......... .......... .......... .......... .......... 75% 96.1M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70200K .......... .......... .......... .......... .......... 75% 120M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70250K .......... .......... .......... .......... .......... 75% 108M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70300K .......... .......... .......... .......... .......... 75% 27.7M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70350K .......... .......... .......... .......... .......... 75% 10.5M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70400K .......... .......... .......... .......... .......... 75% 11.0M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70450K .......... .......... .......... .......... .......... 75% 124M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70500K .......... .......... .......... .......... .......... 75% 116M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70550K .......... .......... .......... .......... .......... 75% 110M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70600K .......... .......... .......... .......... .......... 76% 113M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70650K .......... .......... .......... .......... .......... 76% 116M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70700K .......... .......... .......... .......... .......... 76% 116M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70750K .......... .......... .......... .......... .......... 76% 110M 0s 2023-12-03T11:57:25.686 INFO:teuthology.orchestra.run.smithi116.stderr: 70800K .......... .......... .......... .......... .......... 76% 114M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 70850K .......... .......... .......... .......... .......... 76% 78.0M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 70900K .......... .......... .......... .......... .......... 76% 106M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 70950K .......... .......... .......... .......... .......... 76% 119M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71000K .......... .......... .......... .......... .......... 76% 116M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71050K .......... .......... .......... .......... .......... 76% 107M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71100K .......... .......... .......... .......... .......... 76% 117M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71150K .......... .......... .......... .......... .......... 76% 113M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71200K .......... .......... .......... .......... .......... 76% 116M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71250K .......... .......... .......... .......... .......... 76% 82.1M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71300K .......... .......... .......... .......... .......... 76% 118M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71350K .......... .......... .......... .......... .......... 76% 109M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71400K .......... .......... .......... .......... .......... 76% 120M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71450K .......... .......... .......... .......... .......... 76% 99.2M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71500K .......... .......... .......... .......... .......... 77% 90.2M 0s 2023-12-03T11:57:25.687 INFO:teuthology.orchestra.run.smithi116.stderr: 71550K .......... .......... .......... .......... .......... 77% 105M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71600K .......... .......... .......... .......... .......... 77% 112M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71650K .......... .......... .......... .......... .......... 77% 71.6M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71700K .......... .......... .......... .......... .......... 77% 116M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71750K .......... .......... .......... .......... .......... 77% 121M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71800K .......... .......... .......... .......... .......... 77% 111M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71850K .......... .......... .......... .......... .......... 77% 99.0M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71900K .......... .......... .......... .......... .......... 77% 122M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 71950K .......... .......... .......... .......... .......... 77% 111M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72000K .......... .......... .......... .......... .......... 77% 113M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72050K .......... .......... .......... .......... .......... 77% 11.9M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72100K .......... .......... .......... .......... .......... 77% 81.6M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72150K .......... .......... .......... .......... .......... 77% 115M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72200K .......... .......... .......... .......... .......... 77% 111M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72250K .......... .......... .......... .......... .......... 77% 104M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72300K .......... .......... .......... .......... .......... 77% 117M 0s 2023-12-03T11:57:25.688 INFO:teuthology.orchestra.run.smithi116.stderr: 72350K .......... .......... .......... .......... .......... 77% 121M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72400K .......... .......... .......... .......... .......... 77% 9.92M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72450K .......... .......... .......... .......... .......... 78% 85.3M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72500K .......... .......... .......... .......... .......... 78% 108M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72550K .......... .......... .......... .......... .......... 78% 51.7M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72600K .......... .......... .......... .......... .......... 78% 119M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72650K .......... .......... .......... .......... .......... 78% 102M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72700K .......... .......... .......... .......... .......... 78% 116M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72750K .......... .......... .......... .......... .......... 78% 114M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72800K .......... .......... .......... .......... .......... 78% 109M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72850K .......... .......... .......... .......... .......... 78% 90.0M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72900K .......... .......... .......... .......... .......... 78% 119M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 72950K .......... .......... .......... .......... .......... 78% 105M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 73000K .......... .......... .......... .......... .......... 78% 116M 0s 2023-12-03T11:57:25.689 INFO:teuthology.orchestra.run.smithi116.stderr: 73050K .......... .......... .......... .......... .......... 78% 114M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73100K .......... .......... .......... .......... .......... 78% 111M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73150K .......... .......... .......... .......... .......... 78% 111M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73200K .......... .......... .......... .......... .......... 78% 93.8M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73250K .......... .......... .......... .......... .......... 78% 91.1M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73300K .......... .......... .......... .......... .......... 78% 117M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73350K .......... .......... .......... .......... .......... 79% 113M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73400K .......... .......... .......... .......... .......... 79% 109M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73450K .......... .......... .......... .......... .......... 79% 116M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73500K .......... .......... .......... .......... .......... 79% 117M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73550K .......... .......... .......... .......... .......... 79% 110M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73600K .......... .......... .......... .......... .......... 79% 111M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73650K .......... .......... .......... .......... .......... 79% 79.8M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73700K .......... .......... .......... .......... .......... 79% 131M 0s 2023-12-03T11:57:25.690 INFO:teuthology.orchestra.run.smithi116.stderr: 73750K .......... .......... .......... .......... .......... 79% 103M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 73800K .......... .......... .......... .......... .......... 79% 105M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 73850K .......... .......... .......... .......... .......... 79% 104M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 73900K .......... .......... .......... .......... .......... 79% 65.7M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 73950K .......... .......... .......... .......... .......... 79% 122M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74000K .......... .......... .......... .......... .......... 79% 14.3M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74050K .......... .......... .......... .......... .......... 79% 99.8M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74100K .......... .......... .......... .......... .......... 79% 94.5M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74150K .......... .......... .......... .......... .......... 79% 117M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74200K .......... .......... .......... .......... .......... 79% 120M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74250K .......... .......... .......... .......... .......... 79% 17.7M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74300K .......... .......... .......... .......... .......... 80% 22.4M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74350K .......... .......... .......... .......... .......... 80% 42.8M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74400K .......... .......... .......... .......... .......... 80% 103M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74450K .......... .......... .......... .......... .......... 80% 78.7M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74500K .......... .......... .......... .......... .......... 80% 109M 0s 2023-12-03T11:57:25.691 INFO:teuthology.orchestra.run.smithi116.stderr: 74550K .......... .......... .......... .......... .......... 80% 102M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74600K .......... .......... .......... .......... .......... 80% 113M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74650K .......... .......... .......... .......... .......... 80% 110M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74700K .......... .......... .......... .......... .......... 80% 87.1M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74750K .......... .......... .......... .......... .......... 80% 119M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74800K .......... .......... .......... .......... .......... 80% 97.6M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74850K .......... .......... .......... .......... .......... 80% 76.0M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74900K .......... .......... .......... .......... .......... 80% 110M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 74950K .......... .......... .......... .......... .......... 80% 116M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75000K .......... .......... .......... .......... .......... 80% 103M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75050K .......... .......... .......... .......... .......... 80% 104M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75100K .......... .......... .......... .......... .......... 80% 106M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75150K .......... .......... .......... .......... .......... 80% 108M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75200K .......... .......... .......... .......... .......... 81% 111M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75250K .......... .......... .......... .......... .......... 81% 81.0M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75300K .......... .......... .......... .......... .......... 81% 111M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75350K .......... .......... .......... .......... .......... 81% 90.3M 0s 2023-12-03T11:57:25.692 INFO:teuthology.orchestra.run.smithi116.stderr: 75400K .......... .......... .......... .......... .......... 81% 113M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75450K .......... .......... .......... .......... .......... 81% 102M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75500K .......... .......... .......... .......... .......... 81% 96.3M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75550K .......... .......... .......... .......... .......... 81% 86.3M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75600K .......... .......... .......... .......... .......... 81% 109M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75650K .......... .......... .......... .......... .......... 81% 89.2M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75700K .......... .......... .......... .......... .......... 81% 115M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75750K .......... .......... .......... .......... .......... 81% 112M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75800K .......... .......... .......... .......... .......... 81% 90.9M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75850K .......... .......... .......... .......... .......... 81% 127M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75900K .......... .......... .......... .......... .......... 81% 41.8M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 75950K .......... .......... .......... .......... .......... 81% 33.0M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 76000K .......... .......... .......... .......... .......... 81% 8.84M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 76050K .......... .......... .......... .......... .......... 81% 107M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 76100K .......... .......... .......... .......... .......... 81% 113M 0s 2023-12-03T11:57:25.693 INFO:teuthology.orchestra.run.smithi116.stderr: 76150K .......... .......... .......... .......... .......... 82% 115M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76200K .......... .......... .......... .......... .......... 82% 101M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76250K .......... .......... .......... .......... .......... 82% 97.1M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76300K .......... .......... .......... .......... .......... 82% 133M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76350K .......... .......... .......... .......... .......... 82% 57.1M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76400K .......... .......... .......... .......... .......... 82% 103M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76450K .......... .......... .......... .......... .......... 82% 84.8M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76500K .......... .......... .......... .......... .......... 82% 95.3M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76550K .......... .......... .......... .......... .......... 82% 121M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76600K .......... .......... .......... .......... .......... 82% 115M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76650K .......... .......... .......... .......... .......... 82% 105M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76700K .......... .......... .......... .......... .......... 82% 93.0M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76750K .......... .......... .......... .......... .......... 82% 117M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76800K .......... .......... .......... .......... .......... 82% 122M 0s 2023-12-03T11:57:25.694 INFO:teuthology.orchestra.run.smithi116.stderr: 76850K .......... .......... .......... .......... .......... 82% 85.7M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 76900K .......... .......... .......... .......... .......... 82% 109M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 76950K .......... .......... .......... .......... .......... 82% 106M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77000K .......... .......... .......... .......... .......... 82% 112M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77050K .......... .......... .......... .......... .......... 82% 106M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77100K .......... .......... .......... .......... .......... 83% 88.5M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77150K .......... .......... .......... .......... .......... 83% 105M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77200K .......... .......... .......... .......... .......... 83% 105M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77250K .......... .......... .......... .......... .......... 83% 79.6M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77300K .......... .......... .......... .......... .......... 83% 109M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77350K .......... .......... .......... .......... .......... 83% 116M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77400K .......... .......... .......... .......... .......... 83% 105M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77450K .......... .......... .......... .......... .......... 83% 114M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77500K .......... .......... .......... .......... .......... 83% 99.8M 0s 2023-12-03T11:57:25.695 INFO:teuthology.orchestra.run.smithi116.stderr: 77550K .......... .......... .......... .......... .......... 83% 100M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77600K .......... .......... .......... .......... .......... 83% 117M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77650K .......... .......... .......... .......... .......... 83% 78.3M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77700K .......... .......... .......... .......... .......... 83% 119M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77750K .......... .......... .......... .......... .......... 83% 113M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77800K .......... .......... .......... .......... .......... 83% 116M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77850K .......... .......... .......... .......... .......... 83% 99.6M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77900K .......... .......... .......... .......... .......... 83% 103M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 77950K .......... .......... .......... .......... .......... 83% 115M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78000K .......... .......... .......... .......... .......... 84% 118M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78050K .......... .......... .......... .......... .......... 84% 74.6M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78100K .......... .......... .......... .......... .......... 84% 105M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78150K .......... .......... .......... .......... .......... 84% 98.4M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78200K .......... .......... .......... .......... .......... 84% 94.5M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78250K .......... .......... .......... .......... .......... 84% 35.0M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78300K .......... .......... .......... .......... .......... 84% 69.6M 0s 2023-12-03T11:57:25.696 INFO:teuthology.orchestra.run.smithi116.stderr: 78350K .......... .......... .......... .......... .......... 84% 93.6M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78400K .......... .......... .......... .......... .......... 84% 129M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78450K .......... .......... .......... .......... .......... 84% 87.6M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78500K .......... .......... .......... .......... .......... 84% 93.1M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78550K .......... .......... .......... .......... .......... 84% 75.1M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78600K .......... .......... .......... .......... .......... 84% 57.1M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78650K .......... .......... .......... .......... .......... 84% 50.4M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78700K .......... .......... .......... .......... .......... 84% 103M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78750K .......... .......... .......... .......... .......... 84% 135M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78800K .......... .......... .......... .......... .......... 84% 89.5M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78850K .......... .......... .......... .......... .......... 84% 98.1M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78900K .......... .......... .......... .......... .......... 84% 102M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 78950K .......... .......... .......... .......... .......... 85% 115M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 79000K .......... .......... .......... .......... .......... 85% 107M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 79050K .......... .......... .......... .......... .......... 85% 115M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 79100K .......... .......... .......... .......... .......... 85% 122M 0s 2023-12-03T11:57:25.697 INFO:teuthology.orchestra.run.smithi116.stderr: 79150K .......... .......... .......... .......... .......... 85% 102M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79200K .......... .......... .......... .......... .......... 85% 120M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79250K .......... .......... .......... .......... .......... 85% 86.9M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79300K .......... .......... .......... .......... .......... 85% 93.6M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79350K .......... .......... .......... .......... .......... 85% 125M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79400K .......... .......... .......... .......... .......... 85% 102M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79450K .......... .......... .......... .......... .......... 85% 117M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79500K .......... .......... .......... .......... .......... 85% 90.8M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79550K .......... .......... .......... .......... .......... 85% 70.5M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79600K .......... .......... .......... .......... .......... 85% 100M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79650K .......... .......... .......... .......... .......... 85% 85.7M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79700K .......... .......... .......... .......... .......... 85% 102M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79750K .......... .......... .......... .......... .......... 85% 102M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79800K .......... .......... .......... .......... .......... 85% 118M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79850K .......... .......... .......... .......... .......... 86% 99.6M 0s 2023-12-03T11:57:25.698 INFO:teuthology.orchestra.run.smithi116.stderr: 79900K .......... .......... .......... .......... .......... 86% 93.4M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 79950K .......... .......... .......... .......... .......... 86% 12.4M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80000K .......... .......... .......... .......... .......... 86% 135M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80050K .......... .......... .......... .......... .......... 86% 94.3M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80100K .......... .......... .......... .......... .......... 86% 111M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80150K .......... .......... .......... .......... .......... 86% 70.4M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80200K .......... .......... .......... .......... .......... 86% 99.5M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80250K .......... .......... .......... .......... .......... 86% 116M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80300K .......... .......... .......... .......... .......... 86% 104M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80350K .......... .......... .......... .......... .......... 86% 106M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80400K .......... .......... .......... .......... .......... 86% 66.7M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80450K .......... .......... .......... .......... .......... 86% 111M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80500K .......... .......... .......... .......... .......... 86% 114M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80550K .......... .......... .......... .......... .......... 86% 114M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80600K .......... .......... .......... .......... .......... 86% 118M 0s 2023-12-03T11:57:25.699 INFO:teuthology.orchestra.run.smithi116.stderr: 80650K .......... .......... .......... .......... .......... 86% 95.6M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80700K .......... .......... .......... .......... .......... 86% 115M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80750K .......... .......... .......... .......... .......... 86% 114M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80800K .......... .......... .......... .......... .......... 87% 107M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80850K .......... .......... .......... .......... .......... 87% 81.7M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80900K .......... .......... .......... .......... .......... 87% 103M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 80950K .......... .......... .......... .......... .......... 87% 116M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81000K .......... .......... .......... .......... .......... 87% 110M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81050K .......... .......... .......... .......... .......... 87% 112M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81100K .......... .......... .......... .......... .......... 87% 92.7M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81150K .......... .......... .......... .......... .......... 87% 109M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81200K .......... .......... .......... .......... .......... 87% 118M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81250K .......... .......... .......... .......... .......... 87% 86.7M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81300K .......... .......... .......... .......... .......... 87% 109M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81350K .......... .......... .......... .......... .......... 87% 104M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81400K .......... .......... .......... .......... .......... 87% 109M 0s 2023-12-03T11:57:25.700 INFO:teuthology.orchestra.run.smithi116.stderr: 81450K .......... .......... .......... .......... .......... 87% 105M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81500K .......... .......... .......... .......... .......... 87% 118M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81550K .......... .......... .......... .......... .......... 87% 101M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81600K .......... .......... .......... .......... .......... 87% 103M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81650K .......... .......... .......... .......... .......... 87% 78.9M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81700K .......... .......... .......... .......... .......... 87% 120M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81750K .......... .......... .......... .......... .......... 88% 109M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81800K .......... .......... .......... .......... .......... 88% 113M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81850K .......... .......... .......... .......... .......... 88% 105M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81900K .......... .......... .......... .......... .......... 88% 118M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 81950K .......... .......... .......... .......... .......... 88% 96.8M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82000K .......... .......... .......... .......... .......... 88% 92.3M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82050K .......... .......... .......... .......... .......... 88% 80.8M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82100K .......... .......... .......... .......... .......... 88% 94.9M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82150K .......... .......... .......... .......... .......... 88% 111M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82200K .......... .......... .......... .......... .......... 88% 35.2M 0s 2023-12-03T11:57:25.701 INFO:teuthology.orchestra.run.smithi116.stderr: 82250K .......... .......... .......... .......... .......... 88% 66.5M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82300K .......... .......... .......... .......... .......... 88% 94.7M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82350K .......... .......... .......... .......... .......... 88% 103M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82400K .......... .......... .......... .......... .......... 88% 111M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82450K .......... .......... .......... .......... .......... 88% 82.1M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82500K .......... .......... .......... .......... .......... 88% 102M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82550K .......... .......... .......... .......... .......... 88% 94.5M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82600K .......... .......... .......... .......... .......... 88% 110M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82650K .......... .......... .......... .......... .......... 89% 98.0M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82700K .......... .......... .......... .......... .......... 89% 107M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82750K .......... .......... .......... .......... .......... 89% 91.2M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82800K .......... .......... .......... .......... .......... 89% 118M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82850K .......... .......... .......... .......... .......... 89% 72.9M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82900K .......... .......... .......... .......... .......... 89% 117M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 82950K .......... .......... .......... .......... .......... 89% 108M 0s 2023-12-03T11:57:25.702 INFO:teuthology.orchestra.run.smithi116.stderr: 83000K .......... .......... .......... .......... .......... 89% 107M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83050K .......... .......... .......... .......... .......... 89% 105M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83100K .......... .......... .......... .......... .......... 89% 116M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83150K .......... .......... .......... .......... .......... 89% 102M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83200K .......... .......... .......... .......... .......... 89% 105M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83250K .......... .......... .......... .......... .......... 89% 80.6M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83300K .......... .......... .......... .......... .......... 89% 93.0M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83350K .......... .......... .......... .......... .......... 89% 116M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83400K .......... .......... .......... .......... .......... 89% 109M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83450K .......... .......... .......... .......... .......... 89% 83.5M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83500K .......... .......... .......... .......... .......... 89% 90.0M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83550K .......... .......... .......... .......... .......... 89% 132M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83600K .......... .......... .......... .......... .......... 90% 106M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83650K .......... .......... .......... .......... .......... 90% 80.7M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83700K .......... .......... .......... .......... .......... 90% 110M 0s 2023-12-03T11:57:25.703 INFO:teuthology.orchestra.run.smithi116.stderr: 83750K .......... .......... .......... .......... .......... 90% 102M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 83800K .......... .......... .......... .......... .......... 90% 117M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 83850K .......... .......... .......... .......... .......... 90% 102M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 83900K .......... .......... .......... .......... .......... 90% 67.6M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 83950K .......... .......... .......... .......... .......... 90% 105M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84000K .......... .......... .......... .......... .......... 90% 110M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84050K .......... .......... .......... .......... .......... 90% 62.6M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84100K .......... .......... .......... .......... .......... 90% 104M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84150K .......... .......... .......... .......... .......... 90% 104M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84200K .......... .......... .......... .......... .......... 90% 102M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84250K .......... .......... .......... .......... .......... 90% 117M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84300K .......... .......... .......... .......... .......... 90% 95.1M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84350K .......... .......... .......... .......... .......... 90% 111M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84400K .......... .......... .......... .......... .......... 90% 116M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84450K .......... .......... .......... .......... .......... 90% 96.2M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84500K .......... .......... .......... .......... .......... 91% 114M 0s 2023-12-03T11:57:25.704 INFO:teuthology.orchestra.run.smithi116.stderr: 84550K .......... .......... .......... .......... .......... 91% 91.7M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84600K .......... .......... .......... .......... .......... 91% 114M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84650K .......... .......... .......... .......... .......... 91% 105M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84700K .......... .......... .......... .......... .......... 91% 111M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84750K .......... .......... .......... .......... .......... 91% 96.0M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84800K .......... .......... .......... .......... .......... 91% 110M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84850K .......... .......... .......... .......... .......... 91% 54.4M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84900K .......... .......... .......... .......... .......... 91% 102M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 84950K .......... .......... .......... .......... .......... 91% 113M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85000K .......... .......... .......... .......... .......... 91% 104M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85050K .......... .......... .......... .......... .......... 91% 82.4M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85100K .......... .......... .......... .......... .......... 91% 106M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85150K .......... .......... .......... .......... .......... 91% 127M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85200K .......... .......... .......... .......... .......... 91% 103M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85250K .......... .......... .......... .......... .......... 91% 79.4M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85300K .......... .......... .......... .......... .......... 91% 48.4M 0s 2023-12-03T11:57:25.705 INFO:teuthology.orchestra.run.smithi116.stderr: 85350K .......... .......... .......... .......... .......... 91% 46.5M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85400K .......... .......... .......... .......... .......... 91% 115M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85450K .......... .......... .......... .......... .......... 92% 103M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85500K .......... .......... .......... .......... .......... 92% 73.9M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85550K .......... .......... .......... .......... .......... 92% 161M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85600K .......... .......... .......... .......... .......... 92% 102M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85650K .......... .......... .......... .......... .......... 92% 77.5M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85700K .......... .......... .......... .......... .......... 92% 104M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85750K .......... .......... .......... .......... .......... 92% 117M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85800K .......... .......... .......... .......... .......... 92% 105M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85850K .......... .......... .......... .......... .......... 92% 92.3M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85900K .......... .......... .......... .......... .......... 92% 113M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 85950K .......... .......... .......... .......... .......... 92% 82.6M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 86000K .......... .......... .......... .......... .......... 92% 93.4M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 86050K .......... .......... .......... .......... .......... 92% 66.8M 0s 2023-12-03T11:57:25.706 INFO:teuthology.orchestra.run.smithi116.stderr: 86100K .......... .......... .......... .......... .......... 92% 33.2M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86150K .......... .......... .......... .......... .......... 92% 108M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86200K .......... .......... .......... .......... .......... 92% 25.7M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86250K .......... .......... .......... .......... .......... 92% 80.2M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86300K .......... .......... .......... .......... .......... 92% 100M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86350K .......... .......... .......... .......... .......... 93% 115M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86400K .......... .......... .......... .......... .......... 93% 108M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86450K .......... .......... .......... .......... .......... 93% 69.9M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86500K .......... .......... .......... .......... .......... 93% 100M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86550K .......... .......... .......... .......... .......... 93% 102M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86600K .......... .......... .......... .......... .......... 93% 97.1M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86650K .......... .......... .......... .......... .......... 93% 83.7M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86700K .......... .......... .......... .......... .......... 93% 106M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86750K .......... .......... .......... .......... .......... 93% 90.9M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86800K .......... .......... .......... .......... .......... 93% 112M 0s 2023-12-03T11:57:25.707 INFO:teuthology.orchestra.run.smithi116.stderr: 86850K .......... .......... .......... .......... .......... 93% 83.9M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 86900K .......... .......... .......... .......... .......... 93% 99.6M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 86950K .......... .......... .......... .......... .......... 93% 102M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87000K .......... .......... .......... .......... .......... 93% 103M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87050K .......... .......... .......... .......... .......... 93% 108M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87100K .......... .......... .......... .......... .......... 93% 101M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87150K .......... .......... .......... .......... .......... 93% 113M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87200K .......... .......... .......... .......... .......... 93% 88.7M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87250K .......... .......... .......... .......... .......... 93% 73.6M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87300K .......... .......... .......... .......... .......... 94% 116M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87350K .......... .......... .......... .......... .......... 94% 97.0M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87400K .......... .......... .......... .......... .......... 94% 94.7M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87450K .......... .......... .......... .......... .......... 94% 10.6M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87500K .......... .......... .......... .......... .......... 94% 57.2M 0s 2023-12-03T11:57:25.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87550K .......... .......... .......... .......... .......... 94% 94.5M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87600K .......... .......... .......... .......... .......... 94% 99.3M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87650K .......... .......... .......... .......... .......... 94% 42.9M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87700K .......... .......... .......... .......... .......... 94% 7.68M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87750K .......... .......... .......... .......... .......... 94% 56.3M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87800K .......... .......... .......... .......... .......... 94% 33.3M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87850K .......... .......... .......... .......... .......... 94% 94.5M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87900K .......... .......... .......... .......... .......... 94% 98.3M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87950K .......... .......... .......... .......... .......... 94% 105M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88000K .......... .......... .......... .......... .......... 94% 100M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88050K .......... .......... .......... .......... .......... 94% 85.0M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88100K .......... .......... .......... .......... .......... 94% 104M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88150K .......... .......... .......... .......... .......... 94% 62.4M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88200K .......... .......... .......... .......... .......... 94% 102M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88250K .......... .......... .......... .......... .......... 95% 37.6M 0s 2023-12-03T11:57:25.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88300K .......... .......... .......... .......... .......... 95% 105M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88350K .......... .......... .......... .......... .......... 95% 19.5M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88400K .......... .......... .......... .......... .......... 95% 1.02M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88450K .......... .......... .......... .......... .......... 95% 212M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88500K .......... .......... .......... .......... .......... 95% 245M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88550K .......... .......... .......... .......... .......... 95% 158M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88600K .......... .......... .......... .......... .......... 95% 134M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88650K .......... .......... .......... .......... .......... 95% 123M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88700K .......... .......... .......... .......... .......... 95% 150M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88750K .......... .......... .......... .......... .......... 95% 289M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88800K .......... .......... .......... .......... .......... 95% 254M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88850K .......... .......... .......... .......... .......... 95% 239M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88900K .......... .......... .......... .......... .......... 95% 285M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88950K .......... .......... .......... .......... .......... 95% 310M 0s 2023-12-03T11:57:25.710 INFO:teuthology.orchestra.run.smithi116.stderr: 89000K .......... .......... .......... .......... .......... 95% 339M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89050K .......... .......... .......... .......... .......... 95% 309M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89100K .......... .......... .......... .......... .......... 95% 373M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89150K .......... .......... .......... .......... .......... 96% 389M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89200K .......... .......... .......... .......... .......... 96% 441M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89250K .......... .......... .......... .......... .......... 96% 365M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89300K .......... .......... .......... .......... .......... 96% 397M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89350K .......... .......... .......... .......... .......... 96% 450M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89400K .......... .......... .......... .......... .......... 96% 396M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89450K .......... .......... .......... .......... .......... 96% 405M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89500K .......... .......... .......... .......... .......... 96% 449M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89550K .......... .......... .......... .......... .......... 96% 454M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89600K .......... .......... .......... .......... .......... 96% 352M 0s 2023-12-03T11:57:25.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89650K .......... .......... .......... .......... .......... 96% 26.6M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89700K .......... .......... .......... .......... .......... 96% 81.7M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89750K .......... .......... .......... .......... .......... 96% 211M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89800K .......... .......... .......... .......... .......... 96% 126M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89850K .......... .......... .......... .......... .......... 96% 379M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89900K .......... .......... .......... .......... .......... 96% 438M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89950K .......... .......... .......... .......... .......... 96% 449M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90000K .......... .......... .......... .......... .......... 96% 450M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90050K .......... .......... .......... .......... .......... 96% 367M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90100K .......... .......... .......... .......... .......... 97% 425M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90150K .......... .......... .......... .......... .......... 97% 453M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90200K .......... .......... .......... .......... .......... 97% 357M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90250K .......... .......... .......... .......... .......... 97% 383M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90300K .......... .......... .......... .......... .......... 97% 386M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90350K .......... .......... .......... .......... .......... 97% 415M 0s 2023-12-03T11:57:25.712 INFO:teuthology.orchestra.run.smithi116.stderr: 90400K .......... .......... .......... .......... .......... 97% 448M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90450K .......... .......... .......... .......... .......... 97% 373M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90500K .......... .......... .......... .......... .......... 97% 451M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90550K .......... .......... .......... .......... .......... 97% 460M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90600K .......... .......... .......... .......... .......... 97% 358M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90650K .......... .......... .......... .......... .......... 97% 385M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90700K .......... .......... .......... .......... .......... 97% 382M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90750K .......... .......... .......... .......... .......... 97% 420M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90800K .......... .......... .......... .......... .......... 97% 447M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90850K .......... .......... .......... .......... .......... 97% 359M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90900K .......... .......... .......... .......... .......... 97% 451M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 90950K .......... .......... .......... .......... .......... 97% 449M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 91000K .......... .......... .......... .......... .......... 98% 367M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 91050K .......... .......... .......... .......... .......... 98% 383M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 91100K .......... .......... .......... .......... .......... 98% 421M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 91150K .......... .......... .......... .......... .......... 98% 395M 0s 2023-12-03T11:57:25.713 INFO:teuthology.orchestra.run.smithi116.stderr: 91200K .......... .......... .......... .......... .......... 98% 451M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91250K .......... .......... .......... .......... .......... 98% 370M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91300K .......... .......... .......... .......... .......... 98% 442M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91350K .......... .......... .......... .......... .......... 98% 360M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91400K .......... .......... .......... .......... .......... 98% 8.91M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91450K .......... .......... .......... .......... .......... 98% 378M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91500K .......... .......... .......... .......... .......... 98% 436M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91550K .......... .......... .......... .......... .......... 98% 434M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91600K .......... .......... .......... .......... .......... 98% 422M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91650K .......... .......... .......... .......... .......... 98% 357M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91700K .......... .......... .......... .......... .......... 98% 338M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91750K .......... .......... .......... .......... .......... 98% 407M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91800K .......... .......... .......... .......... .......... 98% 374M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91850K .......... .......... .......... .......... .......... 98% 372M 0s 2023-12-03T11:57:25.714 INFO:teuthology.orchestra.run.smithi116.stderr: 91900K .......... .......... .......... .......... .......... 98% 425M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 91950K .......... .......... .......... .......... .......... 99% 437M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92000K .......... .......... .......... .......... .......... 99% 430M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92050K .......... .......... .......... .......... .......... 99% 358M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92100K .......... .......... .......... .......... .......... 99% 317M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92150K .......... .......... .......... .......... .......... 99% 403M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92200K .......... .......... .......... .......... .......... 99% 376M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92250K .......... .......... .......... .......... .......... 99% 385M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92300K .......... .......... .......... .......... .......... 99% 445M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92350K .......... .......... .......... .......... .......... 99% 435M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92400K .......... .......... .......... .......... .......... 99% 8.59M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92450K .......... .......... .......... .......... .......... 99% 73.7M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92500K .......... .......... .......... .......... .......... 99% 81.5M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92550K .......... .......... .......... .......... .......... 99% 85.1M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92600K .......... .......... .......... .......... .......... 99% 106M 0s 2023-12-03T11:57:25.715 INFO:teuthology.orchestra.run.smithi116.stderr: 92650K .......... .......... .......... .......... .......... 99% 109M 0s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 92700K .......... .......... .......... .......... .......... 99% 104M 0s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 92750K .......... .......... .......... .......... .......... 99% 84.5M 0s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 92800K .......... .......... .......... .......... .......... 99% 108M 0s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 92850K .......... .......... .......... .......... .......... 99% 107M 0s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 92900K 100% 134G=1.1s 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03 11:57:25 (84.6 MB/s) - ‘gradle-6.0.1-bin.zip’ saved [95129672/95129672] 2023-12-03T11:57:25.716 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout:Archive: gradle-6.0.1-bin.zip 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: creating: /opt/gradle/gradle-6.0.1/ 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/getting-started.html 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/LICENSE 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/NOTICE 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: creating: /opt/gradle/gradle-6.0.1/init.d/ 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/init.d/readme.txt 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: creating: /opt/gradle/gradle-6.0.1/bin/ 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle.bat 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: creating: /opt/gradle/gradle-6.0.1/lib/ 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-api-metadata-6.0.1.jar 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-launcher-6.0.1.jar 2023-12-03T11:57:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-runtime-api-info-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-tooling-api-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-wrapper-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-installation-beacon-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-bootstrap-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-file-collections-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-groovy-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-execution-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-core-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-packaging-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-api-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-persistent-cache-6.0.1.jar 2023-12-03T11:57:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-worker-processes-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-logging-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-jvm-services-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-process-services-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-resources-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-messaging-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-native-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-groovy-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-option-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-cli-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-snapshots-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-files-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-tooling-models-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-compiler-embeddable-1.3.50-patched-for-gradle-6.0.1.jar 2023-12-03T11:57:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-hashing-6.0.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-pineapple-6.0.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-docs-6.0.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/groovy-all-1.3-2.5.8.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jul-to-slf4j-1.7.28.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/log4j-over-slf4j-1.7.28.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jcl-over-slf4j-1.7.28.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/slf4j-api-1.7.28.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/guava-27.1-android.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-io-2.6.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-lang-2.6.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-commons-7.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-analysis-7.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-tree-7.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-7.1.jar 2023-12-03T11:57:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-1.10.7.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk8-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/javax.inject-1.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-common-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-host-embeddable-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-embeddable-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-impl-embeddable-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-sam-with-receiver-compiler-plugin-1.3.50.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlinx-metadata-jvm-0.1.0.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-compress-1.19.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/fastutil-8.3.0-min.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jansi-1.18.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jsr305-3.0.2.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kryo-2.24.0.jar 2023-12-03T11:57:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-0.18.jar 2023-12-03T11:57:25.732 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/objenesis-2.6.jar 2023-12-03T11:57:25.732 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/trove4j-1.0.20181211.jar 2023-12-03T11:57:25.732 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/xml-apis-1.4.01.jar 2023-12-03T11:57:25.732 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-reflect-1.3.50.jar 2023-12-03T11:57:25.732 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk7-1.3.50.jar 2023-12-03T11:57:25.733 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-1.3.50.jar 2023-12-03T11:57:25.733 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/annotations-13.0.jar 2023-12-03T11:57:25.733 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-launcher-1.10.7.jar 2023-12-03T11:57:25.733 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/minlog-1.2.jar 2023-12-03T11:57:25.733 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-script-runtime-1.3.50.jar 2023-12-03T11:57:25.738 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-daemon-embeddable-1.3.50.jar 2023-12-03T11:57:25.738 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/failureaccess-1.0.1.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-common-1.3.50.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libcpp-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libstdcpp-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libcpp-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libstdcpp-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-0.18.jar 2023-12-03T11:57:25.741 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-osx-amd64-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-min-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-min-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-ncurses5-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses5-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses6-0.18.jar 2023-12-03T11:57:25.743 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses5-0.18.jar 2023-12-03T11:57:25.744 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses6-0.18.jar 2023-12-03T11:57:25.744 INFO:teuthology.orchestra.run.smithi116.stdout: creating: /opt/gradle/gradle-6.0.1/lib/plugins/ 2023-12-03T11:57:25.744 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-antlr-6.0.1.jar 2023-12-03T11:57:25.744 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-code-quality-6.0.1.jar 2023-12-03T11:57:25.745 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-signing-6.0.1.jar 2023-12-03T11:57:25.745 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-native-6.0.1.jar 2023-12-03T11:57:25.746 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-native-6.0.1.jar 2023-12-03T11:57:25.747 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-native-6.0.1.jar 2023-12-03T11:57:25.749 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-native-6.0.1.jar 2023-12-03T11:57:25.750 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-provider-plugins-6.0.1.jar 2023-12-03T11:57:25.751 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-development-6.0.1.jar 2023-12-03T11:57:25.753 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-maven-6.0.1.jar 2023-12-03T11:57:25.753 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-play-6.0.1.jar 2023-12-03T11:57:25.755 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-6.0.1.jar 2023-12-03T11:57:25.756 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-scala-6.0.1.jar 2023-12-03T11:57:25.756 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ear-6.0.1.jar 2023-12-03T11:57:25.757 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-play-6.0.1.jar 2023-12-03T11:57:25.758 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-javascript-6.0.1.jar 2023-12-03T11:57:25.759 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ivy-6.0.1.jar 2023-12-03T11:57:25.759 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-jacoco-6.0.1.jar 2023-12-03T11:57:25.761 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-init-6.0.1.jar 2023-12-03T11:57:25.762 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-tooling-builders-6.0.1.jar 2023-12-03T11:57:25.764 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugins-6.0.1.jar 2023-12-03T11:57:25.764 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-test-kit-6.0.1.jar 2023-12-03T11:57:25.765 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-groovy-6.0.1.jar 2023-12-03T11:57:25.766 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-scala-6.0.1.jar 2023-12-03T11:57:25.766 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-junit-platform-6.0.1.jar 2023-12-03T11:57:25.766 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-api-builders-6.0.1.jar 2023-12-03T11:57:25.767 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-jvm-6.0.1.jar 2023-12-03T11:57:25.769 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-java-6.0.1.jar 2023-12-03T11:57:25.769 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-jvm-6.0.1.jar 2023-12-03T11:57:25.770 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-jvm-6.0.1.jar 2023-12-03T11:57:25.773 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-native-6.0.1.jar 2023-12-03T11:57:25.775 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-diagnostics-6.0.1.jar 2023-12-03T11:57:25.776 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-publish-6.0.1.jar 2023-12-03T11:57:25.777 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-base-6.0.1.jar 2023-12-03T11:57:25.778 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-base-6.0.1.jar 2023-12-03T11:57:25.779 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-use-6.0.1.jar 2023-12-03T11:57:25.785 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-instant-execution-6.0.1.jar 2023-12-03T11:57:25.785 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-version-control-6.0.1.jar 2023-12-03T11:57:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-dependency-management-6.0.1.jar 2023-12-03T11:57:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-composite-builds-6.0.1.jar 2023-12-03T11:57:25.800 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-reporting-6.0.1.jar 2023-12-03T11:57:25.800 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-s3-6.0.1.jar 2023-12-03T11:57:25.800 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-gcs-6.0.1.jar 2023-12-03T11:57:25.800 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-cache-http-6.0.1.jar 2023-12-03T11:57:25.801 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-http-6.0.1.jar 2023-12-03T11:57:25.801 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-sftp-6.0.1.jar 2023-12-03T11:57:25.802 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-profile-6.0.1.jar 2023-12-03T11:57:25.802 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-workers-6.0.1.jar 2023-12-03T11:57:25.820 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/org.eclipse.jgit-5.5.0.201909110433-r.jar 2023-12-03T11:57:25.820 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-3.0.0.jar 2023-12-03T11:57:25.820 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-shared-3.0.0.jar 2023-12-03T11:57:25.821 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-services-storage-v1-rev136-1.25.0.jar 2023-12-03T11:57:25.822 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-client-1.25.0.jar 2023-12-03T11:57:25.823 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/asm-util-7.1.jar 2023-12-03T11:57:25.831 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ivy-2.3.0.jar 2023-12-03T11:57:25.833 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-compat-3.0.4.jar 2023-12-03T11:57:25.836 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-core-3.0.4.jar 2023-12-03T11:57:25.837 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-common-0.8-20100325.jar 2023-12-03T11:57:25.837 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-groovy-0.8-20100325.jar 2023-12-03T11:57:25.837 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-file-3.0.0.jar 2023-12-03T11:57:25.839 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-container-default-1.7.1.jar 2023-12-03T11:57:25.839 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-connector-wagon-1.13.1.jar 2023-12-03T11:57:25.841 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpg-jdk15on-1.63.jar 2023-12-03T11:57:25.849 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/rhino-1.7.10.jar 2023-12-03T11:57:25.850 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gson-2.8.5.jar 2023-12-03T11:57:25.852 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/simple-4.1.21.jar 2023-12-03T11:57:25.852 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jatl-0.2.3.jar 2023-12-03T11:57:25.853 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-plugin-api-3.0.4.jar 2023-12-03T11:57:25.853 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-oauth-client-1.25.0.jar 2023-12-03T11:57:25.853 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-jackson2-1.25.0.jar 2023-12-03T11:57:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-1.25.0.jar 2023-12-03T11:57:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/testng-6.3.1.jar 2023-12-03T11:57:25.861 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/snakeyaml-1.17.jar 2023-12-03T11:57:25.862 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/dd-plist-1.21.jar 2023-12-03T11:57:25.864 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-4.12.jar 2023-12-03T11:57:25.867 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bsh-2.0b6.jar 2023-12-03T11:57:25.867 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-launcher-1.3.1.jar 2023-12-03T11:57:25.874 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-s3-1.11.633.jar 2023-12-03T11:57:25.877 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-kms-1.11.633.jar 2023-12-03T11:57:25.883 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-core-1.11.633.jar 2023-12-03T11:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpclient-4.5.10.jar 2023-12-03T11:57:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcifs-1.3.17.jar 2023-12-03T11:57:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/nekohtml-1.9.22.jar 2023-12-03T11:57:25.900 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xercesImpl-2.12.0.jar 2023-12-03T11:57:25.902 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jsch-0.1.55.jar 2023-12-03T11:57:25.909 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jaxb-impl-2.3.2.jar 2023-12-03T11:57:25.909 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jmespath-java-1.11.633.jar 2023-12-03T11:57:25.917 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-databind-2.9.10.jar 2023-12-03T11:57:25.920 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-core-2.9.10.jar 2023-12-03T11:57:25.920 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-annotations-2.9.10.jar 2023-12-03T11:57:25.925 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/joda-time-2.10.4.jar 2023-12-03T11:57:25.930 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpkix-jdk15on-1.61.jar 2023-12-03T11:57:25.958 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcprov-jdk15on-1.63.jar 2023-12-03T11:57:25.960 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/commons-codec-1.13.jar 2023-12-03T11:57:25.961 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-aether-provider-3.0.4.jar 2023-12-03T11:57:25.962 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-impl-1.13.1.jar 2023-12-03T11:57:25.963 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-util-1.13.1.jar 2023-12-03T11:57:25.963 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-spi-1.13.1.jar 2023-12-03T11:57:25.964 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-api-1.13.1.jar 2023-12-03T11:57:25.964 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/hamcrest-core-1.3.jar 2023-12-03T11:57:25.966 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpcore-4.4.12.jar 2023-12-03T11:57:25.967 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcommander-1.72.jar 2023-12-03T11:57:25.967 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jzlib-1.1.3.jar 2023-12-03T11:57:25.967 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-artifact-3.0.4.jar 2023-12-03T11:57:25.968 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-builder-3.0.4.jar 2023-12-03T11:57:25.970 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-3.0.4.jar 2023-12-03T11:57:25.970 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-repository-metadata-3.0.4.jar 2023-12-03T11:57:25.971 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-builder-3.0.4.jar 2023-12-03T11:57:25.971 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-3.0.4.jar 2023-12-03T11:57:25.971 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-sec-dispatcher-1.3.jar 2023-12-03T11:57:25.971 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-cipher-1.7.jar 2023-12-03T11:57:25.972 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-classworlds-2.5.1.jar 2023-12-03T11:57:25.972 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-component-annotations-1.5.5.jar 2023-12-03T11:57:25.972 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-interpolation-1.14.jar 2023-12-03T11:57:25.973 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-provider-api-3.0.0.jar 2023-12-03T11:57:25.974 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-utils-3.1.0.jar 2023-12-03T11:57:25.975 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xbean-reflect-3.7.jar 2023-12-03T11:57:25.979 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ion-java-1.0.2.jar 2023-12-03T11:57:25.979 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-engine-1.3.1.jar 2023-12-03T11:57:25.980 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-commons-1.3.1.jar 2023-12-03T11:57:25.981 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/apiguardian-api-1.0.0.jar 2023-12-03T11:57:25.981 INFO:teuthology.orchestra.run.smithi116.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/opentest4j-1.1.1.jar 2023-12-03T11:57:26.020 INFO:teuthology.orchestra.run.smithi116.stdout:export PATH=/opt/gradle/gradle-6.0.1/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2023-12-03T11:57:26.335 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout:Welcome to Gradle 6.0.1! 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout:Here are the highlights of this release: 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Substantial improvements in dependency management, including 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Publishing Gradle Module Metadata in addition to pom.xml 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Advanced control of transitive versions 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Support for optional features and dependencies 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Rules to tweak published metadata 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Support for Java 13 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - Faster incremental Java and Groovy compilation 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - New Zinc compiler for Scala 2023-12-03T11:57:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: - VS2019 support 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: - Support for Gradle Enterprise plugin 3.0 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:For more details see https://docs.gradle.org/6.0.1/release-notes.html 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:------------------------------------------------------------ 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Gradle 6.0.1 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:------------------------------------------------------------ 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Build time: 2019-11-18 20:25:01 UTC 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Revision: fad121066a68c4701acd362daf4287a7c309a0f5 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Kotlin: 1.3.50 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Groovy: 2.5.8 2023-12-03T11:57:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 2023-12-03T11:57:26.338 INFO:teuthology.orchestra.run.smithi116.stdout:JVM: 1.8.0_362 (Red Hat, Inc. 25.362-b08) 2023-12-03T11:57:26.338 INFO:teuthology.orchestra.run.smithi116.stdout:OS: Linux 5.14.0-388.el9.x86_64 amd64 2023-12-03T11:57:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:26.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo keytool -import -alias smithi116.front.sepia.ceph.com -keystore $(readlink -e $(dirname $(readlink -e $(which keytool)))/../lib/security/cacerts) -file /home/ubuntu/cephtest/ca/rgw.client.0.crt -storepass changeit 2023-12-03T11:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:Certificate was added to keystore 2023-12-03T11:57:26.702 DEBUG:tasks.s3tests_java:S3 Tests Java: BEGIN 2023-12-03T11:57:26.702 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-03T11:57:26.703 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T11:57:26.703 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-03T11:57:26.703 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T11:57:26.703 INFO:tasks.s3tests_java:S3 Tests Java: Creating S3 users... 2023-12-03T11:57:27.093 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf is {'bucket_prefix': 'test-', 's3main': {'access_key': None, 'access_secret': None, 'region': 'mexico', 'endpoint': None, 'port': None, 'display_name': None, 'email': None, 'is_secure': None, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data'}} 2023-12-03T11:57:27.093 DEBUG:tasks.s3tests_java:S3 Tests Java: Creating user tester.client.0 2023-12-03T11:57:27.094 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf[s3main] is {'access_key': 'YEQYXJDWATQZCUASSIEY', 'access_secret': '8pxY0nsZk56ypT4WPWWZYxUOMVei5GxuXM4t1W8P7bzyPwbOZx4obQ==', 'region': 'mexico', 'endpoint': 'smithi116.front.sepia.ceph.com:443', 'port': 443, 'display_name': 'Ms. tester.client.0', 'email': 'tester.client.0_test@test.test', 'is_secure': True, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data', 'user_id': 'tester.client.0', 'host': 'smithi116.front.sepia.ceph.com'} 2023-12-03T11:57:27.094 DEBUG:tasks.s3tests_java:S3 Tests Java: Setion, User = s3main, tester.client.0 2023-12-03T11:57:27.094 INFO:tasks.s3tests_java:['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '-n', 'client.0', 'user', 'create', '--uid', 'tester.client.0', '--display-name', 'Ms. tester.client.0', '--access-key', 'YEQYXJDWATQZCUASSIEY', '--secret', '8pxY0nsZk56ypT4WPWWZYxUOMVei5GxuXM4t1W8P7bzyPwbOZx4obQ==', '--email', 'tester.client.0_test@test.test', '--cluster', 'ceph'] 2023-12-03T11:57:27.094 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid tester.client.0 --display-name 'Ms. tester.client.0' --access-key YEQYXJDWATQZCUASSIEY --secret 8pxY0nsZk56ypT4WPWWZYxUOMVei5GxuXM4t1W8P7bzyPwbOZx4obQ== --email tester.client.0_test@test.test --cluster ceph 2023-12-03T11:57:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:57:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.165+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 realm 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.167+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.168+0000 7f96a6523800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.168+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.168+0000 7f96a6523800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:57:27.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.168+0000 7f96a6523800 20 searching for the correct realm 2023-12-03T11:57:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.180+0000 7f96a213c640 0 -- 172.21.15.116:0/1219242322 >> [v2:172.21.15.152:6800/1890087800,v1:172.21.15.152:6801/1890087800] conn(0x56490c7067f0 msgr2=0x56490c708c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:57:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:57:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:57:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:57:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:57:27.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.181+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 zone default found 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 4 Realm: () 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:57:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 zonegroup default 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.182+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.183+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.183+0000 7f96a6523800 20 started sync module instance, tier type = 2023-12-03T11:57:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.183+0000 7f96a6523800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:57:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.189+0000 7f96a293d640 0 -- 172.21.15.116:0/1219242322 >> [v2:172.21.15.152:6808/3202692571,v1:172.21.15.152:6809/3202692571] conn(0x7f9698075510 msgr2=0x7f96980520b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:57:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=0 2023-12-03T11:57:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=1 2023-12-03T11:57:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=2 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=3 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=4 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=5 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=6 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 20 add_watcher() i=7 2023-12-03T11:57:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.192+0000 7f96a6523800 2 all 8 watchers are set, enabling cache 2023-12-03T11:57:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.194+0000 7f9667fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.194+0000 7f96a6523800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:57:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.194+0000 7f96a6523800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:57:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 5 note: GC not initialized 2023-12-03T11:57:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f95d6ffd640 20 reqs_thread_entry: start 2023-12-03T11:57:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 20 init_complete bucket index max shards: 11 2023-12-03T11:57:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 20 Filter name: none 2023-12-03T11:57:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f95d4ff9640 20 reqs_thread_entry: start 2023-12-03T11:57:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2023-12-03T11:57:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 10 cache get: name=default.rgw.meta+users.email+tester.client.0_test@test.test : miss 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.199+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x0 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 adding default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:57:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 cache get: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY : miss 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY info.flags=0x0 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 adding default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY to cache LRU end 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 cache get: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY : hit (negative entry) 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.200+0000 7f96a6523800 10 cache get: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY : hit (negative entry) 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.201+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2023-12-03T11:57:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.201+0000 7f96a6523800 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:57:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.201+0000 7f96a6523800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester.client.0, ofs0, ns] 2023-12-03T11:57:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.202+0000 7f9690ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660547 cookie 94871741333232 notifier 7316 bl.length()=636 2023-12-03T11:57:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.202+0000 7f9690ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2023-12-03T11:57:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.202+0000 7f9690ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f96a6523800 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f96a6523800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:tester.client.0_test@test.test, ofs0, ns] 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f9690ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660505 cookie 94871741286032 notifier 7316 bl.length()=200 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f9690ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2023-12-03T11:57:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.203+0000 7f9690ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:57:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f96a6523800 10 cache put: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY info.flags=0x7 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f96a6523800 10 moving default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY to cache LRU end 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f96a6523800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:YEQYXJDWATQZCUASSIEY, ofs0, ns] 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f9683fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1095216660539 cookie 94871741336352 notifier 7316 bl.length()=189 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f9683fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY info.flags=0x7 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.205+0000 7f9683fff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+YEQYXJDWATQZCUASSIEY to cache LRU end 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stdout: "user_id": "tester.client.0", 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stdout: "display_name": "Ms. tester.client.0", 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stdout: "email": "tester.client.0_test@test.test", 2023-12-03T11:57:27.207 INFO:teuthology.orchestra.run.smithi116.stdout: "suspended": 0, 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max_buckets": 1000, 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "subusers": [], 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "keys": [ 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "user": "tester.client.0", 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "YEQYXJDWATQZCUASSIEY", 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "8pxY0nsZk56ypT4WPWWZYxUOMVei5GxuXM4t1W8P7bzyPwbOZx4obQ==" 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "swift_keys": [], 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": [], 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "op_mask": "read, write, delete", 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_storage_class": "", 2023-12-03T11:57:27.208 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_tags": [], 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "temp_url_keys": [], 2023-12-03T11:57:27.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "rgw", 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stdout: "mfa_ids": [] 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.206+0000 7f9667fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.208+0000 7f96a6523800 20 remove_watcher() i=0 2023-12-03T11:57:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.208+0000 7f96a6523800 2 removed watcher, disabling cache 2023-12-03T11:57:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.209+0000 7f96a6523800 20 remove_watcher() i=1 2023-12-03T11:57:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.211+0000 7f96a6523800 20 remove_watcher() i=2 2023-12-03T11:57:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.212+0000 7f96a6523800 20 remove_watcher() i=3 2023-12-03T11:57:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.213+0000 7f96a6523800 20 remove_watcher() i=4 2023-12-03T11:57:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.214+0000 7f96a6523800 20 remove_watcher() i=5 2023-12-03T11:57:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.215+0000 7f96a6523800 20 remove_watcher() i=6 2023-12-03T11:57:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.216+0000 7f96a6523800 20 remove_watcher() i=7 2023-12-03T11:57:27.223 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-12-03T11:57:27.224 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf 2023-12-03T11:57:27.252 INFO:tasks.s3tests_java:S3 Tests Java: Running tests... 2023-12-03T11:57:27.253 DEBUG:teuthology.orchestra.run.smithi116:> cp /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf /home/ubuntu/cephtest/s3-tests-java/config.properties 2023-12-03T11:57:27.317 DEBUG:teuthology.orchestra.run.smithi116:> radosgw-admin gc process --include-all 2023-12-03T11:57:27.406 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:57:27.406 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:57:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.418+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 realm 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.420+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.421+0000 7f557cd25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.421+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.421+0000 7f557cd25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:57:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.421+0000 7f557cd25800 20 searching for the correct realm 2023-12-03T11:57:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f55701ec640 0 -- 172.21.15.116:0/843929587 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x561a8e543370 msgr2=0x561a8e545760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.432+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 zone default found 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 4 Realm: () 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 zonegroup default 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.433+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.434+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.434+0000 7f557cd25800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.434+0000 7f557cd25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.434+0000 7f557cd25800 20 started sync module instance, tier type = 2023-12-03T11:57:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.434+0000 7f557cd25800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:57:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.442+0000 7f55701ec640 0 -- 172.21.15.116:0/843929587 >> [v2:172.21.15.116:6812/178738993,v1:172.21.15.116:6813/178738993] conn(0x7f5550079e40 msgr2=0x7f555007c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=0 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=1 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=2 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=3 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=4 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=5 2023-12-03T11:57:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=6 2023-12-03T11:57:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 20 add_watcher() i=7 2023-12-03T11:57:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.445+0000 7f557cd25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:57:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.447+0000 7f553f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.447+0000 7f557cd25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:57:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.447+0000 7f557cd25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:57:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.492+0000 7f54ae7fc640 20 reqs_thread_entry: start 2023-12-03T11:57:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.492+0000 7f557cd25800 20 init_complete bucket index max shards: 11 2023-12-03T11:57:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.492+0000 7f557cd25800 20 Filter name: none 2023-12-03T11:57:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.492+0000 7f549ffff640 20 reqs_thread_entry: start 2023-12-03T11:57:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.493+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2023-12-03T11:57:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.494+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.494+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15979362280409482630', time=2023-12-03T13:52:36.282151+0000, chain.objs.size()=1 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.Loa_SWVcAyVTuviRSYt-7CTqdwJgmHC_0 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15921661692034423210', time=2023-12-03T13:52:36.667562+0000, chain.objs.size()=1 2023-12-03T11:57:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.0Z13W0BHcscL3D7gZI-Bj1_DFZ2OXY4_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2198546899682552582', time=2023-12-03T13:54:14.031474+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.iuADZwCxhCgL-0BtkQ4szWIJECtOx8M_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7122597411835208980', time=2023-12-03T13:54:14.608771+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.X_WHScH5UNO7Xe_cGDaKKe2bDOUKDz9_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11475721500202526289', time=2023-12-03T13:54:14.610558+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.3tKU-t5-GdKjZof1AxPvrvXtsSfy5lh_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2761899486657290714', time=2023-12-03T13:54:14.613223+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.dGZWOoeMZlIch58Bl2LM9Wi7xoQD4eu_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14939099312614670978', time=2023-12-03T13:54:45.584775+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.5wSu939aT_bez9SRDdo1-dPFgrQjvOm_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17340377303169313134', time=2023-12-03T13:55:17.577644+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.CvOxt-KOuRE4tfLZr9Ig9GZMEAiSljs_0 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5075854585416460253', time=2023-12-03T13:55:51.365775+0000, chain.objs.size()=1 2023-12-03T11:57:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.8YvBz2IuaF8prs4LObSEJhVzxSaVYy1_0 2023-12-03T11:57:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16475629113613855816', time=2023-12-03T13:56:22.434869+0000, chain.objs.size()=1 2023-12-03T11:57:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.495+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.mY5mp7ZSJfVgxvaqlCv2HKEEk80neCD_0 2023-12-03T11:57:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.501+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.504+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2023-12-03T11:57:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.505+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5277948948846719880', time=2023-12-03T13:52:36.503850+0000, chain.objs.size()=3 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.1 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.3 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2_b5217f9ddabc10ef7ac91bd7af3a8d7b', time=2023-12-03T13:52:36.512208+0000, chain.objs.size()=3 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.1 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.2 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-LUKEWARM.2~qBfoOrjOxDzlqcpeXZ1Iyf5q3nf0KcQ.3 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17234467219421844498', time=2023-12-03T13:54:14.029101+0000, chain.objs.size()=1 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.Wl5QZZQNH6epF2i3FU3vTIZ0mJ_Fo8j_0 2023-12-03T11:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11105087459332197361', time=2023-12-03T13:54:45.589259+0000, chain.objs.size()=1 2023-12-03T11:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.fmH6dLQdsyTZ_IWArOZrlN-yyqdgSGx_0 2023-12-03T11:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6176597848992875767', time=2023-12-03T13:54:46.410064+0000, chain.objs.size()=1 2023-12-03T11:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.506+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.7PdhgSKa_ouHctSQU7FnS2rgXpS_C33_0 2023-12-03T11:57:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.510+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.513+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2023-12-03T11:57:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.514+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.514+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1498367493871780594', time=2023-12-03T13:52:36.264291+0000, chain.objs.size()=1 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.g2NzUYW99OQK7sGVFc2NlUROzKM7Kro_0 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17044074510798205480', time=2023-12-03T13:52:36.631688+0000, chain.objs.size()=1 2023-12-03T11:57:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.HyYv3r4rIXbbhoRKwMLNfakRHPM_qhE_0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16202314418085819159', time=2023-12-03T13:54:14.027144+0000, chain.objs.size()=1 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.7dH9Uk06OXglWVNT-_zCYfRrByK8eht_0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11084835333901411396', time=2023-12-03T13:54:45.590764+0000, chain.objs.size()=1 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.SIBxgE7Rjt8lEqqHo5rE4OPqg-78xGk_0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15774472240759343665', time=2023-12-03T13:54:46.413668+0000, chain.objs.size()=1 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.tCLulhiTPgVZWbmU6IOpHo9atddlx0__0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5578820608561883420', time=2023-12-03T13:54:46.420603+0000, chain.objs.size()=1 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.mn2ct6V-eIVtoylWX8y8-hNw2bjvxwP_0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12360651329928901206', time=2023-12-03T13:55:18.820886+0000, chain.objs.size()=1 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.7DSHSA5qoUr7JVScbIN5lm0GCWlrm6L_0 2023-12-03T11:57:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6482785920959834911', time=2023-12-03T13:55:50.323015+0000, chain.objs.size()=1 2023-12-03T11:57:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.eO0P4GdT15qfuKT3x7hg_DuELxWKHQP_0 2023-12-03T11:57:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4833764314546910556', time=2023-12-03T13:55:50.324924+0000, chain.objs.size()=1 2023-12-03T11:57:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.515+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.DoR6i00mm4Fy1IDT7MtqkL5-iTlkZRy_0 2023-12-03T11:57:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.518+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.521+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2023-12-03T11:57:27.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.522+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.522+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2023-12-03T11:57:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2693513614637591008', time=2023-12-03T13:52:36.268942+0000, chain.objs.size()=1 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.t1Ody72CtrHj6WkIyhQizN9LMJ5OpkX_0 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8116716281473426768', time=2023-12-03T13:52:36.391831+0000, chain.objs.size()=4 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.1 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.2 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.3 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.8__multipart_foo.2~ibx2jA0ju_2eanwbHjcOc1Z8offMBly.4 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14533278622849816686', time=2023-12-03T13:54:46.416615+0000, chain.objs.size()=1 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.odkJdMPiL50DaL2Pmp2tNiQf-O7V-sY_0 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9791131140139457474', time=2023-12-03T13:55:18.830609+0000, chain.objs.size()=1 2023-12-03T11:57:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.pqVe8tabpzjuFG47bAkwfMobg-22Bnn_0 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13631348028476192182', time=2023-12-03T13:55:50.329462+0000, chain.objs.size()=1 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.VGXAf9pV7YmUCf2b81wmRGlP4KAgtHm_0 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7230481374919187920', time=2023-12-03T13:56:22.429881+0000, chain.objs.size()=1 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.5fj636mgsNIGBpCSKOn4SVYBX822Jgo_0 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6118179645423890090', time=2023-12-03T13:56:22.430692+0000, chain.objs.size()=1 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.s5Nsu-cuow3IWDBf1Py6zP5p6vStog5_0 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9838211344481066721', time=2023-12-03T13:56:33.170000+0000, chain.objs.size()=1 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.523+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:PGq831pn78LAqHUULPOE1Wi.p6cdugo_.NmX3SMZBxl7_BE5nSeIKw7XhoYE-m1l_0 2023-12-03T11:57:27.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.528+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.531+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2023-12-03T11:57:27.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.532+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.532+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3049493997185927048', time=2023-12-03T13:52:36.365890+0000, chain.objs.size()=4 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.1 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.2 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.3 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.9__multipart_foo.2~37QBBjKRdg8-Ked9Jv7E9eQ7E0fgPe5.4 2023-12-03T11:57:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='__4_LGsxz9koz2AWclzTLkICdBU2iFvy', time=2023-12-03T13:52:36.516500+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.X2qC_WUVOUwsIkECdGXsnOzX2pfxjrp_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10486094084812017837', time=2023-12-03T13:54:45.595571+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.tLqjyv9xJKbaxnUfTyYYecjoKJKMfBu_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10441153632741908614', time=2023-12-03T13:54:46.413674+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.SSrCtpJf7cK2ex_Tgf1N_b6gi7C2did_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8447826227196507891', time=2023-12-03T13:55:17.574240+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.s9VsCeIxrwCe2YgOfy-_e6dR3u-jQU7_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.18328132351569741398', time=2023-12-03T13:55:18.821725+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.8HDQmgPtUpnDD3aXO5vCA00jBB6orfm_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14714238895298195548', time=2023-12-03T13:55:18.827149+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.BAxgIRiItw2GDH5FHiZyhhQYkXavcSA_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6513764927119035772', time=2023-12-03T13:55:18.830067+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.PGXP_OEDD8ya263txXhCWjyhJ6rU-09_0 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2429436654303743715', time=2023-12-03T13:56:22.423795+0000, chain.objs.size()=1 2023-12-03T11:57:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.533+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.eC4OQL1O0LJh_0Ug7_tYpLXEjIDQwIU_0 2023-12-03T11:57:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.539+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.541+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2023-12-03T11:57:27.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2_384eba532ce0fe32c4793eab7978c65d', time=2023-12-03T13:52:36.508090+0000, chain.objs.size()=3 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.1 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.3 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17751083773362590687', time=2023-12-03T13:54:14.611431+0000, chain.objs.size()=1 2023-12-03T11:57:27.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.r-CJIoracVQBpK8cKeRH852pVTB8qSY_0 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11203379581577257249', time=2023-12-03T13:54:14.615459+0000, chain.objs.size()=1 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.ur1uu1KQJvFRCsjrebwjqlcIKJKdsYM_0 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13897187229829389371', time=2023-12-03T13:54:45.584249+0000, chain.objs.size()=1 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.9rPJqmP4k0Trt67o2RjfYqciGouGOyo_0 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9780550626599417699', time=2023-12-03T13:54:45.589644+0000, chain.objs.size()=1 2023-12-03T11:57:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.543+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.93UIrakaNJ9ez53_Kyf7_H93iIeU0A5_0 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14719032468499651542', time=2023-12-03T13:54:46.413558+0000, chain.objs.size()=1 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.mNYZVUs4n3zIgT2kuH2tdnxFJMFqcLh_0 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11173195058243197327', time=2023-12-03T13:55:18.820701+0000, chain.objs.size()=1 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.D9Wq27gwiv4U4PkT-VKVUYhVbfjTPWr_0 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4448478917475074355', time=2023-12-03T13:55:51.363267+0000, chain.objs.size()=1 2023-12-03T11:57:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.-L69qA1b8Xgzu3eqtOOMi86J5u6BFdN_0 2023-12-03T11:57:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6514333104198922312', time=2023-12-03T13:56:22.878899+0000, chain.objs.size()=1 2023-12-03T11:57:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.544+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.cIhdpidVUt7B3O5AYgVfIEOzjADkrOz_0 2023-12-03T11:57:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.547+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.549+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2023-12-03T11:57:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:57:27.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14838624973424170250', time=2023-12-03T13:52:36.494382+0000, chain.objs.size()=3 2023-12-03T11:57:27.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.1 2023-12-03T11:57:27.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.2 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-STANDARD.2~4OURdTdykcvfAWZ4PerMLGmFZkCcP9d.3 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8997472854241730618', time=2023-12-03T13:54:14.615090+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.9eym1OLTm8UJghkKiv3z6e7TX5EmMOB_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1457348547281620442', time=2023-12-03T13:54:45.585434+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.551+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.rM5vsuENQ1BzV-uXZfeMgn4nFEw0O_O_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13981210472146139452', time=2023-12-03T13:54:46.411335+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.Ocw8tjzAOm-I8xhmbSOgKz_jmQIlzaq_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1497487016538499723', time=2023-12-03T13:55:18.818340+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.crYgOAXgIaeCg6VlfO90lYezlfzK84t_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17201184703344100098', time=2023-12-03T13:55:50.331602+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.RLd2WLVomusbEkkHbSHGGWVKgZsmxdV_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12888972564504389475', time=2023-12-03T13:55:50.335209+0000, chain.objs.size()=1 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.552+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.NUrTcK8PShAw5u9p2Lyp6pJoEgpO9EQ_0 2023-12-03T11:57:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.553+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10398341499650870837', time=2023-12-03T13:56:22.878916+0000, chain.objs.size()=1 2023-12-03T11:57:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.553+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.kZBdRTZj2UZsoyth6YI_kRvaqnWirqM_0 2023-12-03T11:57:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.553+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6895558419734044192', time=2023-12-03T13:56:33.170418+0000, chain.objs.size()=1 2023-12-03T11:57:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.553+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:ro2QzucJviHMLEp9sIMwWiZm8VOybbj_.work0UQK6byA_64IwzTcKqp9wshE6Ea_0 2023-12-03T11:57:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.555+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.558+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2023-12-03T11:57:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.559+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_1_29a088c966b4426474263a92ed1bbae5', time=2023-12-03T13:52:36.455354+0000, chain.objs.size()=1 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_0 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6062969768876879931', time=2023-12-03T13:52:36.676199+0000, chain.objs.size()=1 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.84b-a49TxSI-75YaRNe343zGi1uEtqH_0 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3870609225289665052', time=2023-12-03T13:54:14.027538+0000, chain.objs.size()=1 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.NsETAvrAIoIERLuwuluxMntylSVsiP7_0 2023-12-03T11:57:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12816452797062404933', time=2023-12-03T13:54:14.031657+0000, chain.objs.size()=1 2023-12-03T11:57:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.Kc7pJefm3ZmgiF9KxtjUOstuI2pqExk_0 2023-12-03T11:57:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11442243703318571981', time=2023-12-03T13:54:14.610016+0000, chain.objs.size()=1 2023-12-03T11:57:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.560+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.NgiJkgtaVo79X8axP4PXP8MmiogukgP_0 2023-12-03T11:57:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.562+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.564+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2023-12-03T11:57:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.566+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.566+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4771697587165576162', time=2023-12-03T13:52:36.635656+0000, chain.objs.size()=1 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.LNNaVLlCDXO5Xh4QvVIyVM4LqS9prsA_0 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2075766675150422150', time=2023-12-03T13:54:46.416146+0000, chain.objs.size()=1 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.odRog3U7eygkxw9x1gYIJOzhU-xfnjR_0 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3945753468067915319', time=2023-12-03T13:55:17.577602+0000, chain.objs.size()=1 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.9oXvbd_OgMhsw9HYQ7dBZ0i9lsUG3Zj_0 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6049217725735126793', time=2023-12-03T13:55:50.324178+0000, chain.objs.size()=1 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.A6gwNFoNdinHs_czNr79kqHpi7EqRe3_0 2023-12-03T11:57:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16882232049746269853', time=2023-12-03T13:55:50.329371+0000, chain.objs.size()=1 2023-12-03T11:57:27.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.CO1cmHhEgMBqo--nIT2QstrOsL5uM3i_0 2023-12-03T11:57:27.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6554277533904853519', time=2023-12-03T13:55:51.363425+0000, chain.objs.size()=1 2023-12-03T11:57:27.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.osDOMNITfk6_PpInXH0FyaoJ0PfTs2S_0 2023-12-03T11:57:27.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6833647072600486160', time=2023-12-03T13:56:22.878487+0000, chain.objs.size()=1 2023-12-03T11:57:27.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.567+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.mzyueHyZpIj99hxgitUPJukY6gvm66J_0 2023-12-03T11:57:27.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.570+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.572+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2023-12-03T11:57:27.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.573+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:57:27.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_1_dd6586667d2712fc49bc888a04884fd4', time=2023-12-03T13:52:36.447565+0000, chain.objs.size()=1 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_0 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_zbP6jHYOyjuM6RaZ_dt6BUc5aXQ5ZFu', time=2023-12-03T13:52:36.499975+0000, chain.objs.size()=2 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.x7P7Zzlo3ljalaSx1Ya3QyyAtgHrZk4_0 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.x7P7Zzlo3ljalaSx1Ya3QyyAtgHrZk4_1 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16633758735362359023', time=2023-12-03T13:52:36.583321+0000, chain.objs.size()=2 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.9SYsYG0saUoeEHdHv2f2V9DwlXRseAt_0 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.9SYsYG0saUoeEHdHv2f2V9DwlXRseAt_1 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4379356959610052139', time=2023-12-03T13:54:46.415667+0000, chain.objs.size()=1 2023-12-03T11:57:27.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.0qj2H6ej2ImbrSnL1Z1aEfOmXYJc2kr_0 2023-12-03T11:57:27.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9953658086895814121', time=2023-12-03T13:55:18.820290+0000, chain.objs.size()=1 2023-12-03T11:57:27.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.ZX1D7CTQhpQ3qjT21sCDXAfY5FntbVi_0 2023-12-03T11:57:27.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16025916433040108191', time=2023-12-03T13:55:51.355127+0000, chain.objs.size()=1 2023-12-03T11:57:27.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.574+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.NYl__5xh4Hxfks334iIFBsmqiwynCMd_0 2023-12-03T11:57:27.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.577+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.580+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2023-12-03T11:57:27.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.581+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.581+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=13, truncated=0, next_marker='' 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_fgeJ7KqvwsZAAvugDm7o6b-2qtDZ3m3', time=2023-12-03T13:52:36.490644+0000, chain.objs.size()=2 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.839ojSGzlETQ10xxfERISGGAiPfrERd_0 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.839ojSGzlETQ10xxfERISGGAiPfrERd_1 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14405823307011223176', time=2023-12-03T13:54:14.034827+0000, chain.objs.size()=1 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.4yrXjDhXVoV-4qmW-88z7QhDw2BMpPx_0 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16263615810665661183', time=2023-12-03T13:54:14.620913+0000, chain.objs.size()=1 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.9-xb2BAz0rrdtL4jsflaM7ToTMQj40I_0 2023-12-03T11:57:27.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15855866184453793777', time=2023-12-03T13:54:45.596078+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.iOz0IkO5y_ywgE-SjgmrJr0cvU_O4K6_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6982147869257268749', time=2023-12-03T13:54:46.410633+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.nNPbDi-q63pg0BQWNvf3jDoU5ybsQqW_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.628456307349406345', time=2023-12-03T13:54:46.421293+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.UNrHEPfzqhSOOyja1wPPpJCR4wWHhDF_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13548973340652293823', time=2023-12-03T13:55:18.819894+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.vVfZKlBQKqcSrqh3w8SgMYxF_EMaieD_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6943281497663813484', time=2023-12-03T13:55:18.822493+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.SFydxlkOxfSK1mH2BnwoMWPez5VkQHN_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10326737389894605018', time=2023-12-03T13:55:19.420474+0000, chain.objs.size()=1 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.cNJNIfIBVnqlsjYZHlA2QRF9cAiQ4lr_0 2023-12-03T11:57:27.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2680395369747083537', time=2023-12-03T13:55:19.423447+0000, chain.objs.size()=1 2023-12-03T11:57:27.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.dPSoM4eugR7eCbn9cQIA9w-Y7UdZCFx_0 2023-12-03T11:57:27.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9231018201480785059', time=2023-12-03T13:55:19.432450+0000, chain.objs.size()=1 2023-12-03T11:57:27.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.582+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.Xq0c-3HFFgHxQMCP5q6CfnwpB1HE2g6_0 2023-12-03T11:57:27.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.585+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4341954161115394470', time=2023-12-03T13:55:51.356083+0000, chain.objs.size()=1 2023-12-03T11:57:27.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.585+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.XelcnRNXgN1qCwP_dVvbptt3sfiTOwT_0 2023-12-03T11:57:27.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.585+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3760053449516934908', time=2023-12-03T13:56:22.434422+0000, chain.objs.size()=1 2023-12-03T11:57:27.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.585+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.C9LxmlIeO0B9NRGiBdMPHRT3zsa_s9P_0 2023-12-03T11:57:27.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.587+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.589+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2023-12-03T11:57:27.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.591+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.591+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3764564422396261561', time=2023-12-03T13:52:36.286315+0000, chain.objs.size()=1 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.RcpctYp5LKtZFsjuxL5jVxBg8PsKn7L_0 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16621818171130358970', time=2023-12-03T13:54:14.028098+0000, chain.objs.size()=1 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.UnbbFJHL44H0vchqFxobbfk2Xx-d9m3_0 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12401919476766661996', time=2023-12-03T13:54:46.418724+0000, chain.objs.size()=1 2023-12-03T11:57:27.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.pjnsKEedaGEBS7AuYo866Q8ASgUEuLr_0 2023-12-03T11:57:27.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16755402839812282558', time=2023-12-03T13:55:50.329372+0000, chain.objs.size()=1 2023-12-03T11:57:27.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.592+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.zwSmOU7SFOevdxjMVnBwiP9o8Qe-lv5_0 2023-12-03T11:57:27.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.593+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.596+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2023-12-03T11:57:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.597+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.597+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=13, truncated=0, next_marker='' 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1381388202791168885', time=2023-12-03T13:52:36.299530+0000, chain.objs.size()=1 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.CYb1UMSe9teQYCFEj5H40m28QDm3RDS_0 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5029240034335594067', time=2023-12-03T13:52:36.546836+0000, chain.objs.size()=3 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.1 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.2 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-STANDARD.2~94rCIgl_Z9Dmtp8cPGmEERgW6FBJ3RK.3 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14316874995745245393', time=2023-12-03T13:54:45.583988+0000, chain.objs.size()=1 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.b8yapE4rjADx1Qhf0aUsWNuK-3b6Mq0_0 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7833334398028683156', time=2023-12-03T13:54:46.411681+0000, chain.objs.size()=1 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.QpPdyQi44sFRL4-XTreX2RdqFk7a7YL_0 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15073352708642691472', time=2023-12-03T13:55:17.575989+0000, chain.objs.size()=1 2023-12-03T11:57:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.9s4ForlwpPNlNxO3L-Wz5WiALzvJrN9_0 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15611329685080578261', time=2023-12-03T13:55:17.580465+0000, chain.objs.size()=1 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.baWDarCwJENlWF1rHeCla07tIEgNRHW_0 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15647418982103078074', time=2023-12-03T13:55:18.826226+0000, chain.objs.size()=1 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.hd6ds3R5HJRqKRZlblrSWNIHG_gLmVS_0 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11700193892104633702', time=2023-12-03T13:55:19.421394+0000, chain.objs.size()=1 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.P-rRHouHoyGid6n76B72ZqtxqTCF8ZW_0 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8912606331447046079', time=2023-12-03T13:55:19.430871+0000, chain.objs.size()=1 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.LywamstLq0AfRzd9WycGZLj-G2kuNWT_0 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12004552710300607086', time=2023-12-03T13:55:50.330769+0000, chain.objs.size()=1 2023-12-03T11:57:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.598+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.NRV7UKVv9u2d6isAWJcYiFqGDmOy-K9_0 2023-12-03T11:57:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.600+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13493424137558671522', time=2023-12-03T13:55:50.335856+0000, chain.objs.size()=1 2023-12-03T11:57:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.600+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.4rhxps0iRwZMAKmQv5WonhDZ1P4HsNZ_0 2023-12-03T11:57:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.600+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13436169883530296781', time=2023-12-03T13:56:22.425666+0000, chain.objs.size()=1 2023-12-03T11:57:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.600+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.DhAIBexqPFvYVD6FmFdf6_vQlmjqLVt_0 2023-12-03T11:57:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.603+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9894046115742444409', time=2023-12-03T13:56:22.877611+0000, chain.objs.size()=1 2023-12-03T11:57:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.603+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.D7rFpCy99neEqyg98AsGd3o1eLMwGoa_0 2023-12-03T11:57:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.607+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.609+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2023-12-03T11:57:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=14, truncated=0, next_marker='' 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17074618796293171486', time=2023-12-03T13:52:36.273692+0000, chain.objs.size()=1 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.1dxWJshyR-U6-Av3RLfwHcklVA4-F2N_0 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17432873725421458260', time=2023-12-03T13:52:36.340759+0000, chain.objs.size()=4 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.1 2023-12-03T11:57:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.2 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.611+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.3 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.11__multipart_foo.2~w4o1McLB7gtqerzD_fjnVik8CNY2EPy.4 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_lYvrtSJgl4-HEaHIeFnEQMNqtH9-9JY', time=2023-12-03T13:52:36.478481+0000, chain.objs.size()=2 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.nmchWCxC0NP7hiJT0NJEhWJnJpOYggE_0 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.nmchWCxC0NP7hiJT0NJEhWJnJpOYggE_1 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2088851970941462436', time=2023-12-03T13:54:14.025482+0000, chain.objs.size()=1 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.CcsXU5PndTZ41zf6ovOek6N_7r-_PDD_0 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11667336211746522192', time=2023-12-03T13:54:14.611692+0000, chain.objs.size()=1 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.ShIX4V6DZj-t6npw4Trw8qcmxrlcZbn_0 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14170928629402469834', time=2023-12-03T13:54:45.587368+0000, chain.objs.size()=1 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.ym5LOnwAFgreccGCGoX2cxxHe_5AptP_0 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15951764210916184874', time=2023-12-03T13:55:18.351419+0000, chain.objs.size()=1 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.ItcQ1ZyautVf3FUoKWKqKqMLLnVt4fr_0 2023-12-03T11:57:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.704900526641944668', time=2023-12-03T13:55:19.421689+0000, chain.objs.size()=1 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.612+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.kFnx_Sdz1FbJQCqB0_sX9WMd8PaihaN_0 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8445016590076166794', time=2023-12-03T13:55:50.324251+0000, chain.objs.size()=1 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.IatFUxvqCnlRjviqm6QgzTIo_BP0oRI_0 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4511602962409509617', time=2023-12-03T13:55:50.329040+0000, chain.objs.size()=1 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.miaOr8D0o04XBLybICUrGE977c1GT0L_0 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15475566254781425904', time=2023-12-03T13:55:50.329859+0000, chain.objs.size()=1 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.614+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.goGH7B4RzMGJ33LUTVjI6qGxWL9b92k_0 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.615+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10532300587184292234', time=2023-12-03T13:55:50.331058+0000, chain.objs.size()=1 2023-12-03T11:57:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.615+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.mBumarqhtsCAKXNSS2fp4E8lKlms4Hz_0 2023-12-03T11:57:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.615+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2410269665146943586', time=2023-12-03T13:55:51.357654+0000, chain.objs.size()=1 2023-12-03T11:57:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.615+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.ITlidd0dF9OHMLVAln3yAk6RpfiRX3Y_0 2023-12-03T11:57:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.616+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.18405548405012348471', time=2023-12-03T13:56:22.428581+0000, chain.objs.size()=1 2023-12-03T11:57:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.616+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.4dey7zvX4d6HsoQMzwUW3-JwVHKyPBL_0 2023-12-03T11:57:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.618+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.621+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2023-12-03T11:57:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.622+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.622+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:57:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7795625664471973848', time=2023-12-03T13:54:14.041013+0000, chain.objs.size()=1 2023-12-03T11:57:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.B8mvAH0SXkqppFLQmYuaeoAqaTjPyhr_0 2023-12-03T11:57:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.21682755872668153', time=2023-12-03T13:55:17.575050+0000, chain.objs.size()=1 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_._5rA1-JH8lOrHqphsxnAoM9nGFzwzwh_0 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13483243411292997651', time=2023-12-03T13:55:17.576893+0000, chain.objs.size()=1 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.eoEzBzZjgU0cj97tHgr19QB0cjDq5eQ_0 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5586925270977008694', time=2023-12-03T13:55:19.425469+0000, chain.objs.size()=1 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.T4YXvoXImRfRKtUdX5cKMorPYrZkazu_0 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6774455453185820280', time=2023-12-03T13:55:51.354526+0000, chain.objs.size()=1 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.UthCHhhzqthkEFosd7zkLb7IKGv_de-_0 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1591189028365005843', time=2023-12-03T13:55:51.356823+0000, chain.objs.size()=1 2023-12-03T11:57:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.623+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.SKbQxeZ89FkTYVHUmMOzarxyP0Tyhpz_0 2023-12-03T11:57:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.626+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.628+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2023-12-03T11:57:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.629+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:57:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2489060453889625357', time=2023-12-03T13:52:36.295216+0000, chain.objs.size()=1 2023-12-03T11:57:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.nuPbOHk_FQWz0497UOPBt3ak91Wyoxq_0 2023-12-03T11:57:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2_37bc1b9a1cecd79a1d831445f9312037', time=2023-12-03T13:52:36.523441+0000, chain.objs.size()=3 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.3 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15361356955815292459', time=2023-12-03T13:54:14.615320+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.POo17-vejtqb0bzGSDBMJdaAi5WaU6e_0 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5350931446801330950', time=2023-12-03T13:54:14.622643+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.ijPjnjv2x_00C31kzXNjMbhGrhpQSJP_0 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.18241869569077941742', time=2023-12-03T13:55:17.574586+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.VgUKqwLu3m3f4fnpEykWxoRvyz24RRj_0 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12424978559237223714', time=2023-12-03T13:55:17.576303+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.FE8g6kX8BnCJTl0fC5OBlvBsF1oUwbB_0 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6795205356336625733', time=2023-12-03T13:55:18.821710+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.0etjyyi2uYdfw-8jTdRATHfImlVsFYm_0 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16329462235482031246', time=2023-12-03T13:55:19.431012+0000, chain.objs.size()=1 2023-12-03T11:57:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.tj1reLK2Tl-WN4VwWSqSPziZAKFgkwi_0 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8684499922098149964', time=2023-12-03T13:56:22.425713+0000, chain.objs.size()=1 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.H4qcN225Xvt6es-0Cs-VG_SK1WXQHXH_0 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14914478417489555114', time=2023-12-03T13:56:22.878856+0000, chain.objs.size()=1 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.630+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.Sf_qhlzai_j4GyFK0E40XkWzm-kzepS_0 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.633+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6135402291965444089', time=2023-12-03T13:56:22.880220+0000, chain.objs.size()=1 2023-12-03T11:57:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.633+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.D5sHtJmQrnyjrJ_Gf3mBR8VVZdWlvCn_0 2023-12-03T11:57:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.634+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.637+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2023-12-03T11:57:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.638+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.638+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2023-12-03T11:57:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_cgZJFTbstzVeMT30xjIt9JtgL3AuRIO', time=2023-12-03T13:52:36.451552+0000, chain.objs.size()=1 2023-12-03T11:57:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._DBUQ0bbe12kEPgYG2zxEdmfGJ4Bvkx_0 2023-12-03T11:57:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16086371989616476687', time=2023-12-03T13:55:19.422013+0000, chain.objs.size()=1 2023-12-03T11:57:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.grhO39OKSAP5s4L4wIc9eJhxWCsmmoC_0 2023-12-03T11:57:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13126579936768017782', time=2023-12-03T13:55:19.426620+0000, chain.objs.size()=1 2023-12-03T11:57:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.hNZ2T6IdVF9cSfOWmZcVSGhuzS1f2d0_0 2023-12-03T11:57:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10514859022299042156', time=2023-12-03T13:56:22.425540+0000, chain.objs.size()=1 2023-12-03T11:57:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.639+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.3toHtaOpxqom0pO_g9tczS4S3VzbU5D_0 2023-12-03T11:57:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.640+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.642+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2023-12-03T11:57:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_eEw5OutN_ftEL2VNNLtwTzFwIRhoZxV', time=2023-12-03T13:52:36.435460+0000, chain.objs.size()=1 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.G__hUT1GN-xsq-p6ifMOPZAeQaSr9uo_0 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_W1cRGeeexJOhhMeffN3mBDikz8eor_w', time=2023-12-03T13:52:36.474650+0000, chain.objs.size()=1 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.GnBocMFTZPOZhJY6ggTAFLyRjYqfGIQ_0 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16055724886094821947', time=2023-12-03T13:54:45.586375+0000, chain.objs.size()=1 2023-12-03T11:57:27.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.k85K3rdRKFE4vY9V_syebDtWtwFCLQB_0 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16144132388026669416', time=2023-12-03T13:54:45.589153+0000, chain.objs.size()=1 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.l2TSijIA_YBeiqu2pXxbNUvLdR-cVnp_0 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1074782091943493790', time=2023-12-03T13:55:17.586208+0000, chain.objs.size()=1 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.q1FelhKI1k0P3NbkiVX806UgXwdS1tY_0 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4615393561994770450', time=2023-12-03T13:55:51.355666+0000, chain.objs.size()=1 2023-12-03T11:57:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.644+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.VsWk6NIIOF7jlipoVXUmeZm4eVybKsV_0 2023-12-03T11:57:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.647+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.649+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2023-12-03T11:57:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.650+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_gv7vn5ZihUg5Jj52XiO7wWpUi7eUvNr', time=2023-12-03T13:52:36.427632+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.dR2zcFiFsWjFQHQZj3d9mEPPTth_dFI_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6449173615384990142', time=2023-12-03T13:54:14.026296+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.gm81XkihsRql9reyQuQeaQaywlTYeBV_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3535837683918977801', time=2023-12-03T13:54:14.028637+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.Pzooek28txti8SVR4h7xUOkuS0JOfLw_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11428546382783378995', time=2023-12-03T13:54:14.035697+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.AJsk0A54sCvVGNk7CeeJwk1_OSYQ2kH_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12167171582961371657', time=2023-12-03T13:54:14.612407+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.9p0XT2Np3R0AgCKVliNBXSAx0sNxEB-_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8549834358455652021', time=2023-12-03T13:54:45.586564+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.CAPCDAszc3Xn_Hf7GRq-D8XiYgICwNI_0 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8347641383952091087', time=2023-12-03T13:55:17.575431+0000, chain.objs.size()=1 2023-12-03T11:57:27.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.sQrMHbfS4dmejuDsqwRH-idnCK5vfIb_0 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15379009955005143105', time=2023-12-03T13:55:18.829066+0000, chain.objs.size()=1 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.9GDTaLAr4ylMBmby7ZOlIDzqZSXEVrX_0 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3784043140571352329', time=2023-12-03T13:55:19.431808+0000, chain.objs.size()=1 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.651+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.kr4EanSRBG63aRwdVPkaNFSd6NhN4xh_0 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.652+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13484918807394011420', time=2023-12-03T13:56:22.435963+0000, chain.objs.size()=1 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.652+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.BKbiYA26xrqISOsCtqHM8bB6fYnywji_0 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.652+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15134697061969112567', time=2023-12-03T13:56:22.882298+0000, chain.objs.size()=1 2023-12-03T11:57:27.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.652+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.KazimNXk2ra-CrEppA5WbOIQgjpGNDW_0 2023-12-03T11:57:27.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.656+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.658+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2023-12-03T11:57:27.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.659+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15437882696372543921', time=2023-12-03T13:52:36.643769+0000, chain.objs.size()=1 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.mVeNYCiMspEMDAHlkZ02aOgz9YNTaj3_0 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14207444108627386568', time=2023-12-03T13:54:14.027621+0000, chain.objs.size()=1 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.Xl9cGA5HdTWzO64wzMjzcGMpVqwUirq_0 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8637413030440113269', time=2023-12-03T13:54:45.595220+0000, chain.objs.size()=1 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.wZ8OER0x6LRIjwcCo5KuJ0wKeR3QhdI_0 2023-12-03T11:57:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10752261916012037162', time=2023-12-03T13:54:46.416514+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.VAf5i9hBbU9EKYL7jknyi1E3oGphZsK_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6353130400711972621', time=2023-12-03T13:55:17.576101+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.0_EcX7IVaAGeNiGBbEDteInNr91DSA2_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3767773211130834474', time=2023-12-03T13:55:19.424654+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.alqTgcJ1vTbvjmKWwEBt62exjoXjLkQ_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4801488157984000981', time=2023-12-03T13:55:51.358802+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.F3CrbzXCc1zpjquYRDiAr1N_yVvO3Xx_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11633551339477775702', time=2023-12-03T13:55:51.359765+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.6K2F-clZysw3Vd1ohcLaY8J-C_u_eK7_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6927062266413990053', time=2023-12-03T13:56:21.958487+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.zLTUrjchp0_Ln4XFOgtpsTHKyQdv_Hg_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1972280193841829719', time=2023-12-03T13:56:22.427709+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_._ZmvQfI-b-DClBKylpXhm3kIfpeIJNB_0 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5481258934973473723', time=2023-12-03T13:56:33.170399+0000, chain.objs.size()=1 2023-12-03T11:57:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.660+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:wy928QuYFRLXVfaS-Bml9pyyNTVVtG7_.fHBkIewFsddsCCGgow4BrTyb4rtEwp7_0 2023-12-03T11:57:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.664+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.666+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2023-12-03T11:57:27.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.668+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.668+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3825308987436127956', time=2023-12-03T13:52:36.551179+0000, chain.objs.size()=3 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.1 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.2 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-LUKEWARM.2~M-zCTyzU4qsLDAjIfliEfgOlZaR8yoo.3 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7235507170229646328', time=2023-12-03T13:52:36.573797+0000, chain.objs.size()=2 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.MNPLcAiQl4YbnSvl82nKaqLqkXtOv1y_0 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.MNPLcAiQl4YbnSvl82nKaqLqkXtOv1y_1 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1825559174693096223', time=2023-12-03T13:54:14.032048+0000, chain.objs.size()=1 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_._O5Xp0BJUY1e_HG_3Xyn-MVsZXiaUba_0 2023-12-03T11:57:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2522564476049111701', time=2023-12-03T13:54:46.146852+0000, chain.objs.size()=1 2023-12-03T11:57:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.ey0rEmlpl2ufwz8YCcMLZkHWREicfG6_0 2023-12-03T11:57:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6177438698500297429', time=2023-12-03T13:55:51.359739+0000, chain.objs.size()=1 2023-12-03T11:57:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.669+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.t2JeYNe32IRpxvIuSKi-rKTXDkyCvhc_0 2023-12-03T11:57:27.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.673+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.675+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2023-12-03T11:57:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2023-12-03T11:57:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7395227752656352250', time=2023-12-03T13:52:36.290495+0000, chain.objs.size()=1 2023-12-03T11:57:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.UokA8FBegyNtaR_C1v0ljw7ixbwMH0f_0 2023-12-03T11:57:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4518676667839594598', time=2023-12-03T13:52:36.419855+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.rxV5JoPA8mfDy9gRZO7hagvOh9S6tMU_0 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2791064542133733893', time=2023-12-03T13:52:36.482491+0000, chain.objs.size()=3 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.2 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__multipart_foo-FROZEN.2~mTxf9BYUCaMG9dE64-P4Pm1s7J_mFwH.3 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5860484723574191926', time=2023-12-03T13:54:14.608744+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.USSGEEe7U0_Cyxlhmwcpx-Psu-EHYO4_0 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17230726924325137279', time=2023-12-03T13:54:14.612349+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.z6gSBykQKN30lZnsb3r8bHYy1l-Hg9H_0 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5573094646498679380', time=2023-12-03T13:54:15.184933+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.kptsprzx08bntvhkIdbScajJviLRNU0_0 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10921500325674549491', time=2023-12-03T13:55:19.422047+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.-tsiX8FWwRyV14Y_f1rIeBdtxxaCoiF_0 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3200817477028319515', time=2023-12-03T13:55:19.424796+0000, chain.objs.size()=1 2023-12-03T11:57:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.677+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.pU81L-ExaMSGBkC8dr7nVfTRN4oiNBi_0 2023-12-03T11:57:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.680+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.683+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2023-12-03T11:57:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.684+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.685+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.685+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.685+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7027472809808533510', time=2023-12-03T13:52:36.439966+0000, chain.objs.size()=1 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.685+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.f4SIl4763p_sKM0LHb1BiluuiM-uYKR_0 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4800958086152152473', time=2023-12-03T13:52:36.577718+0000, chain.objs.size()=1 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.12__shadow_.Z24apEM09jSDV0bi6uoGQVgG34UlKbJ_0 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8887994219598782985', time=2023-12-03T13:54:45.584061+0000, chain.objs.size()=1 2023-12-03T11:57:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.JSeuMsLlNA1i4IKe50WbW1Gbfn0NZ67_0 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9365003486372249409', time=2023-12-03T13:55:17.580276+0000, chain.objs.size()=1 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.oHrLaxuArlBkiQrrppgTYUr8F_su3si_0 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11094174746799582563', time=2023-12-03T13:55:19.426684+0000, chain.objs.size()=1 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.jXoBm4x7j064zuPN-4JHPcu_l6RzzmB_0 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3694907245140804996', time=2023-12-03T13:56:22.426263+0000, chain.objs.size()=1 2023-12-03T11:57:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.686+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.iqlHobf_2EXTLE7heqiLnsB6j-vn-fn_0 2023-12-03T11:57:27.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.689+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.691+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2023-12-03T11:57:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.692+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_an27X9ERsVYN0rlUmvADL6QHH5Mv4q0', time=2023-12-03T13:52:36.415603+0000, chain.objs.size()=1 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._O2YUxZhBMtuQXPCF5im355p1gFn450_0 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9935991511208099609', time=2023-12-03T13:54:14.033058+0000, chain.objs.size()=1 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.SLbZyrcrZ5BBRqQdbIZFEIEOsWxmgQa_0 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5108301975075490236', time=2023-12-03T13:54:14.616055+0000, chain.objs.size()=1 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.xSzI8c5eATO624hTBePIyYnUh6srAWg_0 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1434344563230181345', time=2023-12-03T13:54:14.617263+0000, chain.objs.size()=1 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.1obvGe0EkqKe-38JgspawuJCcc54Xp1_0 2023-12-03T11:57:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12647283472489482420', time=2023-12-03T13:54:14.620541+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.4e_HnsYlSDmX-Rq_nJlVTvnE0MpcupS_0 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6056186532210584083', time=2023-12-03T13:54:46.409987+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.u0opTttrNLV0hYp-rzoeNjI-J8usdx0_0 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.18328282156906358578', time=2023-12-03T13:55:18.823636+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.la1OqqEF2gx2jtBRCUoCtrzpjbZpFBi_0 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15342246495312899843', time=2023-12-03T13:55:19.426113+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.DhOv5kNDjs02lVtK7FmxB9IJkBKeQRi_0 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3468014699685532090', time=2023-12-03T13:55:51.356786+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.CYWeDlxT7eKalADf1Auqyb7dMgCgFps_0 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4782141603429305998', time=2023-12-03T13:56:22.424232+0000, chain.objs.size()=1 2023-12-03T11:57:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.693+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.t8bxdQIhd9djy2hHaT8CJmox6uVfDRL_0 2023-12-03T11:57:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.696+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.699+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2023-12-03T11:57:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15983392924009970420', time=2023-12-03T13:54:14.024405+0000, chain.objs.size()=1 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.7Zgxn2nHZeshX2vsgNr6i8MRu8K_sOl_0 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14988221899409740776', time=2023-12-03T13:54:14.027339+0000, chain.objs.size()=1 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.6ncMf7qM_bYzogC5jl5jLzxXpKg5ckX_0 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1128902058323044706', time=2023-12-03T13:55:18.824109+0000, chain.objs.size()=1 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.ZJjrL2XuoUVneBG4hqxPL8lPE0NAhEK_0 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16054612543137613860', time=2023-12-03T13:55:51.354999+0000, chain.objs.size()=1 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.Ono4edr0F50-FjXhmerWPVX--_liqIc_0 2023-12-03T11:57:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17850168398437357042', time=2023-12-03T13:56:22.429958+0000, chain.objs.size()=1 2023-12-03T11:57:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.701+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.HnDmEMAmfzgGlAaHvP5Nwo0k1AVt5S3_0 2023-12-03T11:57:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.704+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.706+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2023-12-03T11:57:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.707+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_EXfnec-TpQfJhzhej3L61fCVNxPumD0', time=2023-12-03T13:52:36.411809+0000, chain.objs.size()=1 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.UlJBo2shC-YiPmV44NA7hWVRdT5Mtku_0 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.456617908150793397', time=2023-12-03T13:52:36.639624+0000, chain.objs.size()=1 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.6__shadow_.Ie5LuIlHKy0Et9w2FhG1PMMQeMZxNBw_0 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12618120339025927870', time=2023-12-03T13:54:45.586860+0000, chain.objs.size()=1 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.n-rVV7xLKOebU6OtVc1jsocwYGjAcXD_0 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3301264955283154119', time=2023-12-03T13:55:18.820645+0000, chain.objs.size()=1 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.8u9O-P928PKPRFYnuUqtHnewZ-RT5IR_0 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.10325287964528758174', time=2023-12-03T13:55:18.822518+0000, chain.objs.size()=1 2023-12-03T11:57:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.c2oEum_a7MDW8CcYzkIhnTdkGSbZa_3_0 2023-12-03T11:57:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.583043858302182287', time=2023-12-03T13:55:51.355863+0000, chain.objs.size()=1 2023-12-03T11:57:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.Tjql7IvwYn5Creom7LoKtb-d1gHugck_0 2023-12-03T11:57:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17671430303622882056', time=2023-12-03T13:56:22.427294+0000, chain.objs.size()=1 2023-12-03T11:57:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.708+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.C0ySdQucitMBw526-Ip1kj5lx5WpxpD_0 2023-12-03T11:57:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.711+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.713+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2023-12-03T11:57:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.714+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:57:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2409511898137047588', time=2023-12-03T13:52:36.542768+0000, chain.objs.size()=3 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.1 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.2 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:21f2df23-11b5-4c91-821f-a8e55156195e.4668.13__multipart_foo-FROZEN.2~M-w1sUOfqnhktWdiTYByyLVg3UPAjcM.3 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.321573323563310060', time=2023-12-03T13:52:36.671570+0000, chain.objs.size()=1 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.-1tlZPjr8xlamemArrtRxoCZMtoiCmB_0 2023-12-03T11:57:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.403563675378618656', time=2023-12-03T13:54:14.026710+0000, chain.objs.size()=1 2023-12-03T11:57:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.wq5NclUxbOSmIM9sN4SWaWcbwG_oyJK_0 2023-12-03T11:57:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1283253479444327985', time=2023-12-03T13:54:45.584053+0000, chain.objs.size()=1 2023-12-03T11:57:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.LNjCypc-t1aoct9DDEo433abZZNgJ2j_0 2023-12-03T11:57:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9656481940450411391', time=2023-12-03T13:55:17.576157+0000, chain.objs.size()=1 2023-12-03T11:57:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.OVX8POj_T7Z9xyN6OPQhlPKJpxhJe8s_0 2023-12-03T11:57:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6107636197741669095', time=2023-12-03T13:55:19.421584+0000, chain.objs.size()=1 2023-12-03T11:57:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.5Ms8u6fE-blIYQDIiKrq-kOm20GN0vw_0 2023-12-03T11:57:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4997025060060607268', time=2023-12-03T13:56:29.613639+0000, chain.objs.size()=1 2023-12-03T11:57:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.715+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.17__shadow:qWi4VSU6W5b-TG9UGIv3itfxXiivvs._.farF_wu_EyilYKxBeAGZ2GcprVYtu3L_0 2023-12-03T11:57:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.718+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.722+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2023-12-03T11:57:27.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.724+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15253833040509796658', time=2023-12-03T13:52:36.278111+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.3__shadow_.t_aCElvapfCgA3hayMTTfNuyQ7BJUbp_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_AV9vx8GvxkaSHP5Akw_NcvEi37-JKQ4', time=2023-12-03T13:52:36.423659+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_._F8XjXqNnKGgNXmI8YRqT9YpgP_wG4c_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.657222621384221037', time=2023-12-03T13:54:46.413680+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.QCiU9inCExPcPVVBD1-liIiFSJRugMc_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16935234613657528608', time=2023-12-03T13:54:46.418741+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.8vvhenLrqDOxrlqvZpWOxEPGcpcrGkT_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13379924731374835346', time=2023-12-03T13:54:46.421377+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_.a96QeHiYH1xLpSdltgEae8GWqwBOr3J_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8456673634855843622', time=2023-12-03T13:55:17.577146+0000, chain.objs.size()=1 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.67Q4N8lruGtpy1ePxviZG7fsTlPVQmt_0 2023-12-03T11:57:27.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17822396428652359446', time=2023-12-03T13:55:17.579892+0000, chain.objs.size()=1 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.QnCU0KQFdaOco5sj3FH_8FEftcIrYLE_0 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17575917452841774651', time=2023-12-03T13:55:18.820923+0000, chain.objs.size()=1 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.eToM8_7XQrUrqOrLHdYmoYW-Cf2sTrj_0 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1006133692288398934', time=2023-12-03T13:55:19.424815+0000, chain.objs.size()=1 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.Z_rfAnPyT7hXhyY3UzfX7pPokyyt7Wm_0 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1797412564864037353', time=2023-12-03T13:55:50.325660+0000, chain.objs.size()=1 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.bVBYR2DH54wAvzKbcueTlTCVw7OTmd2_0 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8760003957102312657', time=2023-12-03T13:55:51.365039+0000, chain.objs.size()=1 2023-12-03T11:57:27.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.725+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.irE3C2E18RJJjMRTaBgQtY4Db_g4NQE_0 2023-12-03T11:57:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.729+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.731+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2023-12-03T11:57:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.732+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.732+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.2351895489632245315', time=2023-12-03T13:52:36.663457+0000, chain.objs.size()=1 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.7__shadow_.6-JA0l1lh4phThqUZ_vQXabD8nDLGLK_0 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8106705308676077994', time=2023-12-03T13:55:19.423426+0000, chain.objs.size()=1 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.ytvkRrD_Sy8v3QDqWgbRSzDvuG4B7NC_0 2023-12-03T11:57:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12825527146127129151', time=2023-12-03T13:55:50.323238+0000, chain.objs.size()=1 2023-12-03T11:57:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.8vVX37gqfxl-e8p-ItwFsrZOZ2_YVvY_0 2023-12-03T11:57:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16093403007032556934', time=2023-12-03T13:55:50.325594+0000, chain.objs.size()=1 2023-12-03T11:57:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.8y7JS_WZTGmey9tTfF8mLMvc4WUYouN_0 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8560259206130338379', time=2023-12-03T13:55:51.093213+0000, chain.objs.size()=1 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.OA-qJgBTGMEAYqw-jxs1kKjtZywLRd4_0 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7676840192365243411', time=2023-12-03T13:55:51.357526+0000, chain.objs.size()=1 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.iuKFQMiEWwad91U7IlWbdDZcHyjMst6_0 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16668151541386895248', time=2023-12-03T13:55:51.360678+0000, chain.objs.size()=1 2023-12-03T11:57:27.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.733+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.PgRKwCW6qpQs_UaN5Li6wy_lSYq0eRu_0 2023-12-03T11:57:27.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.736+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.738+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2023-12-03T11:57:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.739+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2023-12-03T11:57:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5645470404707769389', time=2023-12-03T13:52:36.431760+0000, chain.objs.size()=1 2023-12-03T11:57:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_0 2023-12-03T11:57:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_1_5bb6526ccfc1a4edf41357788adf66e4', time=2023-12-03T13:52:36.443681+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.14__shadow_.Ryp9Ka9NtFW1uJz7zNGr9q7TwCnBrQ5_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3112057701269628767', time=2023-12-03T13:54:13.762099+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.ZMQzv979c543Jp7Rk3yq5DbwXRpw8c4_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9409208357259659383', time=2023-12-03T13:54:14.609582+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.1WKtwZiJ1hncBr-lPexEk_E2u98k-3F_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.6507729014167932983', time=2023-12-03T13:54:45.586810+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.ObRxX6XAkg-tZACmx_KZSF6K9hkPfyt_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.75384086366360378', time=2023-12-03T13:54:45.588458+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.20__shadow_.ro8Sz425tXpAW7YiCs9VIEYKKk8w2dp_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.4361614452601303647', time=2023-12-03T13:54:47.179540+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.vfctJQOKXF6ejyB9d6rqR4z_sRB1SbP_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.12085949757133228589', time=2023-12-03T13:55:18.817926+0000, chain.objs.size()=1 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.23__shadow_.S9zNPznrhf-FZWzcarno6W0xHmf09uo_0 2023-12-03T11:57:27.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9925288025349619695', time=2023-12-03T13:55:19.423291+0000, chain.objs.size()=1 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.24__shadow_.VH18i-3XliYpyF233W7s8EkW86BTP-q_0 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.7035509632684331708', time=2023-12-03T13:55:51.360342+0000, chain.objs.size()=1 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.26__shadow_.OQ2hrzMC1FcH9uuUXn544RqLP5_rn1H_0 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9901420097570887374', time=2023-12-03T13:56:22.421790+0000, chain.objs.size()=1 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.DainNt2MfdRjhxuThps0jrAUrjwZXdF_0 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16255522972388046933', time=2023-12-03T13:56:22.424441+0000, chain.objs.size()=1 2023-12-03T11:57:27.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.740+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.xCiZ_S0NqUv8PAJgZZcA3WKZPnmagEs_0 2023-12-03T11:57:27.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.744+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.747+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2023-12-03T11:57:27.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.748+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.748+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.748+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.748+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='_evG-0XTX0c6DpgAWHxuk4_q0PeBiVb3', time=2023-12-03T13:52:36.486898+0000, chain.objs.size()=2 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.748+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.Pnv1TDUX2p6_zuRydL7PO2c-zx2pecR_0 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:21f2df23-11b5-4c91-821f-a8e55156195e.4668.15__shadow_.Pnv1TDUX2p6_zuRydL7PO2c-zx2pecR_1 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.15106844612457121537', time=2023-12-03T13:54:14.611783+0000, chain.objs.size()=1 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.I2iBUQxf5U2DzQ_J8JD3oB1r8zCjfbF_0 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.1880198647202250081', time=2023-12-03T13:54:14.620886+0000, chain.objs.size()=1 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.19__shadow_.0fKs2mVaYwsQORFlx_qAdpOO6HLdapL_0 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.3874363453369311335', time=2023-12-03T13:55:17.584393+0000, chain.objs.size()=1 2023-12-03T11:57:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.GiGlBaJMDjoGXqeZBgISYTYmgqtJYsy_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.5981440526211355848', time=2023-12-03T13:55:17.586066+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_.H2FMzwofgKsw39510YiG1uxfFJiscw-_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16740271386261968541', time=2023-12-03T13:55:50.045822+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.-hH7xXe5xn9Ngog4PN0kfYx39se_eZf_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.11463364872790227045', time=2023-12-03T13:55:50.330189+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.cb2mqV9DKMLKyDkdg39pN-caKsp3kyq_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14641535619647501250', time=2023-12-03T13:55:50.334432+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.25__shadow_.RW2aQwZBtnfyd7vaDNMnmyBDSV9haG6_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.14613765479532872804', time=2023-12-03T13:56:22.425701+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.rrGNu_fDZ9PlnwMpe49zoQlz1INawsL_0 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8912745186678836606', time=2023-12-03T13:56:22.428350+0000, chain.objs.size()=1 2023-12-03T11:57:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.749+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.27__shadow_.ld7il2jbbjSIMlh5JtAxU2C-9qHkKZT_0 2023-12-03T11:57:27.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.753+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.755+0000 7f557cd25800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2023-12-03T11:57:27.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.756+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.756+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.16943246894898149922', time=2023-12-03T13:52:36.319874+0000, chain.objs.size()=4 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.1 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.2 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.3 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.10__multipart_foo.2~M5MOnsgefsMfzFnlXzdzxHUkklHXtPq.4 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.9986834263046126867', time=2023-12-03T13:54:14.029821+0000, chain.objs.size()=1 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.18__shadow_.oGUUMAOo8YGPhCewhaEvmGqy4ZAL2r5_0 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.13935561997766627523', time=2023-12-03T13:54:46.413700+0000, chain.objs.size()=1 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.21__shadow_._ZpxblBBvaFmDCjaVRYDmck-RiOk3Fz_0 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.17127345728521270885', time=2023-12-03T13:55:17.576818+0000, chain.objs.size()=1 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.22__shadow_._M3crvrK3WFs8VpVEpRBI1qZF7D1mPg_0 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 20 garbage collection: RGWGC::process iterating over entry tag='21f2df23-11b5-4c91-821f-a8e55156195e.4662.8823205668538131117', time=2023-12-03T13:56:22.880116+0000, chain.objs.size()=1 2023-12-03T11:57:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.757+0000 7f557cd25800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:21f2df23-11b5-4c91-821f-a8e55156195e.4668.16__shadow_.BqCjnO0FbY3tTEBXS6FMHyw8yLFX78z_0 2023-12-03T11:57:27.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.760+0000 7f557cd25800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:57:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.764+0000 7f553f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:27.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.766+0000 7f557cd25800 20 remove_watcher() i=0 2023-12-03T11:57:27.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.766+0000 7f557cd25800 2 removed watcher, disabling cache 2023-12-03T11:57:27.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.767+0000 7f557cd25800 20 remove_watcher() i=1 2023-12-03T11:57:27.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.767+0000 7f557cd25800 20 remove_watcher() i=2 2023-12-03T11:57:27.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.769+0000 7f557cd25800 20 remove_watcher() i=3 2023-12-03T11:57:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.770+0000 7f557cd25800 20 remove_watcher() i=4 2023-12-03T11:57:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.770+0000 7f557cd25800 20 remove_watcher() i=5 2023-12-03T11:57:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.771+0000 7f557cd25800 20 remove_watcher() i=6 2023-12-03T11:57:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.772+0000 7f557cd25800 20 remove_watcher() i=7 2023-12-03T11:57:27.779 DEBUG:teuthology.orchestra.run.smithi116:> radosgw-admin gc process --include-all 2023-12-03T11:57:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:57:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:57:27.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.839+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 20 realm 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.841+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.842+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.842+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.842+0000 7f2a7172c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.842+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.843+0000 7f2a7172c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:57:27.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.843+0000 7f2a7172c800 20 searching for the correct realm 2023-12-03T11:57:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.851+0000 7f2a6d4f0640 0 -- 172.21.15.116:0/370416708 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2a6802a970 msgr2=0x7f2a6802d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:57:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 RGWRados::pool_iterate: got zonegroup_info.f0b0f691-5884-41b8-97f7-3e766649f55a 2023-12-03T11:57:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 RGWRados::pool_iterate: got zone_info.21f2df23-11b5-4c91-821f-a8e55156195e 2023-12-03T11:57:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:57:27.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 zone default found 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 4 Realm: () 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 4 ZoneGroup: default (f0b0f691-5884-41b8-97f7-3e766649f55a) 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 4 Zone: default (21f2df23-11b5-4c91-821f-a8e55156195e) 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 zonegroup default 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.853+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 rados->read ofs=0 len=0 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 started sync module instance, tier type = 2023-12-03T11:57:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.854+0000 7f2a7172c800 20 started zone id=21f2df23-11b5-4c91-821f-a8e55156195e (name=default) with tier type = 2023-12-03T11:57:27.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.860+0000 7f2a6d4f0640 0 -- 172.21.15.116:0/370416708 >> [v2:172.21.15.116:6824/4019364149,v1:172.21.15.116:6825/4019364149] conn(0x7f2a48079d90 msgr2=0x7f2a4807c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:57:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=0 2023-12-03T11:57:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=1 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=2 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=3 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=4 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=5 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.863+0000 7f2a7172c800 20 add_watcher() i=6 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.864+0000 7f2a7172c800 20 add_watcher() i=7 2023-12-03T11:57:27.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.864+0000 7f2a7172c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:57:27.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.865+0000 7f2a32fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:27.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.866+0000 7f2a7172c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:57:27.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.866+0000 7f2a7172c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:57:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.904+0000 7f29a1ffb640 20 reqs_thread_entry: start 2023-12-03T11:57:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.904+0000 7f2a7172c800 20 init_complete bucket index max shards: 11 2023-12-03T11:57:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.904+0000 7f2a7172c800 20 Filter name: none 2023-12-03T11:57:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.904+0000 7f29937fe640 20 reqs_thread_entry: start 2023-12-03T11:57:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.904+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2023-12-03T11:57:27.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.906+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.906+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.907+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.908+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2023-12-03T11:57:27.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.909+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.909+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.910+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.911+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2023-12-03T11:57:27.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.912+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.913+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.913+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.914+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2023-12-03T11:57:27.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.915+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.916+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.916+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.917+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2023-12-03T11:57:27.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.918+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.919+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.919+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.920+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2023-12-03T11:57:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.921+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.922+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.922+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.923+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2023-12-03T11:57:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.924+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.925+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.925+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.926+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2023-12-03T11:57:27.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.928+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.928+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.929+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.930+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2023-12-03T11:57:27.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.931+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.932+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.932+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.933+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2023-12-03T11:57:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.934+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.934+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.935+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.935+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2023-12-03T11:57:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.937+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.937+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.937+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.938+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2023-12-03T11:57:27.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.939+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.940+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.940+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.941+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2023-12-03T11:57:27.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.942+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.943+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.943+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.944+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2023-12-03T11:57:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.945+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.946+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.946+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.948+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2023-12-03T11:57:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.949+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.950+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.950+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.951+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2023-12-03T11:57:27.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.953+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.953+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.953+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.954+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2023-12-03T11:57:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.956+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.956+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.956+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.957+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2023-12-03T11:57:27.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.959+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.959+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.960+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.961+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2023-12-03T11:57:27.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.962+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.963+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.963+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.964+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2023-12-03T11:57:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.965+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.966+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.966+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.967+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2023-12-03T11:57:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.969+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.969+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.969+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.970+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2023-12-03T11:57:27.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.972+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.972+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.973+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.974+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2023-12-03T11:57:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.975+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.975+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.976+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.977+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2023-12-03T11:57:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.978+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.979+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.979+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.980+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2023-12-03T11:57:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.981+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.981+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.981+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.982+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2023-12-03T11:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.984+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.984+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.984+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.985+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2023-12-03T11:57:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.987+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.987+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.987+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.988+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2023-12-03T11:57:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.989+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.990+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.990+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.991+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2023-12-03T11:57:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.993+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.993+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.994+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.994+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2023-12-03T11:57:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.996+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.996+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:27.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.997+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.998+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2023-12-03T11:57:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.999+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.999+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:27.999+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.000+0000 7f2a7172c800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2023-12-03T11:57:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.001+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.002+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:57:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.002+0000 7f2a7172c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:57:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.004+0000 7f2a32fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:57:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.006+0000 7f2a7172c800 20 remove_watcher() i=0 2023-12-03T11:57:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.006+0000 7f2a7172c800 2 removed watcher, disabling cache 2023-12-03T11:57:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.007+0000 7f2a7172c800 20 remove_watcher() i=1 2023-12-03T11:57:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.008+0000 7f2a7172c800 20 remove_watcher() i=2 2023-12-03T11:57:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.009+0000 7f2a7172c800 20 remove_watcher() i=3 2023-12-03T11:57:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.010+0000 7f2a7172c800 20 remove_watcher() i=4 2023-12-03T11:57:28.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.011+0000 7f2a7172c800 20 remove_watcher() i=5 2023-12-03T11:57:28.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.012+0000 7f2a7172c800 20 remove_watcher() i=6 2023-12-03T11:57:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T11:57:28.013+0000 7f2a7172c800 20 remove_watcher() i=7 2023-12-03T11:57:28.020 DEBUG:teuthology.orchestra.run.smithi116:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test 2023-12-03T11:57:28.495 INFO:teuthology.orchestra.run.smithi116.stdout:Starting a Gradle Daemon (subsequent builds will be faster) 2023-12-03T11:57:31.996 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :clean UP-TO-DATE 2023-12-03T11:57:53.395 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :compileJava 2023-12-03T11:57:53.396 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :processResources 2023-12-03T11:57:53.396 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :classes 2023-12-03T11:57:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:53.996 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :compileTestJava 2023-12-03T11:57:53.996 INFO:teuthology.orchestra.run.smithi116.stderr:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2023-12-03T11:57:53.996 INFO:teuthology.orchestra.run.smithi116.stderr:Note: Recompile with -Xlint:deprecation for details. 2023-12-03T11:57:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:53.997 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :processTestResources NO-SOURCE 2023-12-03T11:57:53.997 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :testClasses 2023-12-03T11:57:59.896 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :test 2023-12-03T11:57:59.896 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:57:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 STARTED 2023-12-03T11:58:00.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:00.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 PASSED 2023-12-03T11:58:00.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 STARTED 2023-12-03T11:58:00.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 PASSED 2023-12-03T11:58:00.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:00.995 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 STARTED 2023-12-03T11:58:01.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 PASSED 2023-12-03T11:58:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.695 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 STARTED 2023-12-03T11:58:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.695 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 PASSED 2023-12-03T11:58:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 STARTED 2023-12-03T11:58:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 PASSED 2023-12-03T11:58:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:01.995 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 STARTED 2023-12-03T11:58:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:02.395 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 PASSED 2023-12-03T11:58:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:02.695 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STARTED 2023-12-03T11:58:06.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:06.995 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 PASSED 2023-12-03T11:58:07.295 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:07.295 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 STARTED 2023-12-03T11:58:07.295 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:07.295 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 PASSED 2023-12-03T11:58:07.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:07.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 STARTED 2023-12-03T11:58:07.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:07.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 PASSED 2023-12-03T11:58:07.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:07.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 STARTED 2023-12-03T11:58:08.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:08.195 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 PASSED 2023-12-03T11:58:08.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:08.396 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 STARTED 2023-12-03T11:58:08.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:08.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 PASSED 2023-12-03T11:58:08.595 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:08.595 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 STARTED 2023-12-03T11:58:10.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:10.995 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 PASSED 2023-12-03T11:58:11.295 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:11.295 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 STARTED 2023-12-03T11:58:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:11.795 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 PASSED 2023-12-03T11:58:12.095 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:12.095 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 STARTED 2023-12-03T11:58:12.295 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:12.295 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 PASSED 2023-12-03T11:58:12.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:12.495 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 STARTED 2023-12-03T11:58:19.895 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:19.895 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 PASSED 2023-12-03T11:58:20.095 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:20.095 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 STARTED 2023-12-03T11:58:21.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:21.195 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 PASSED 2023-12-03T11:58:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:21.395 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 STARTED 2023-12-03T11:58:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:22.395 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED 2023-12-03T11:58:22.595 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:22.595 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED 2023-12-03T11:58:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:22.896 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 PASSED 2023-12-03T11:58:23.106 INFO:tasks.rgw.client.0.smithi116.stdout:*** Caught signal (Segmentation fault) ** 2023-12-03T11:58:23.107 INFO:tasks.rgw.client.0.smithi116.stdout: in thread 7f74a3217640 thread_name:radosgw 2023-12-03T11:58:23.107 INFO:tasks.rgw.client.0.smithi116.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:58:23.107 INFO:tasks.rgw.client.0.smithi116.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f74e9654db0] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x5596ef08d66a] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x5596eef22fb1] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x5596eef26a75] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 5: radosgw(+0x39f71d) [0x5596eee4771d] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 6: radosgw(+0x3cf019) [0x5596eee77019] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 7: make_fcontext() 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout:2023-12-03T11:58:23.106+0000 7f74a3217640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: in thread 7f74a3217640 thread_name:radosgw 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f74e9654db0] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x5596ef08d66a] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x5596eef22fb1] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x5596eef26a75] 2023-12-03T11:58:23.108 INFO:tasks.rgw.client.0.smithi116.stdout: 5: radosgw(+0x39f71d) [0x5596eee4771d] 2023-12-03T11:58:23.109 INFO:tasks.rgw.client.0.smithi116.stdout: 6: radosgw(+0x3cf019) [0x5596eee77019] 2023-12-03T11:58:23.109 INFO:tasks.rgw.client.0.smithi116.stdout: 7: make_fcontext() 2023-12-03T11:58:23.109 INFO:tasks.rgw.client.0.smithi116.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:58:23.109 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 0> 2023-12-03T11:58:23.106+0000 7f74a3217640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: in thread 7f74a3217640 thread_name:radosgw 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f74e9654db0] 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x5596ef08d66a] 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x5596eef22fb1] 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x5596eef26a75] 2023-12-03T11:58:23.120 INFO:tasks.rgw.client.0.smithi116.stdout: 5: radosgw(+0x39f71d) [0x5596eee4771d] 2023-12-03T11:58:23.121 INFO:tasks.rgw.client.0.smithi116.stdout: 6: radosgw(+0x3cf019) [0x5596eee77019] 2023-12-03T11:58:23.121 INFO:tasks.rgw.client.0.smithi116.stdout: 7: make_fcontext() 2023-12-03T11:58:23.121 INFO:tasks.rgw.client.0.smithi116.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:58:23.121 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.132 INFO:tasks.rgw.client.0.smithi116.stdout: -99> 2023-12-03T11:58:23.106+0000 7f74a3217640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:58:23.132 INFO:tasks.rgw.client.0.smithi116.stdout: in thread 7f74a3217640 thread_name:radosgw 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f74e9654db0] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x5596ef08d66a] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x5596eef22fb1] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x5596eef26a75] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 5: radosgw(+0x39f71d) [0x5596eee4771d] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 6: radosgw(+0x3cf019) [0x5596eee77019] 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 7: make_fcontext() 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:58:23.133 INFO:tasks.rgw.client.0.smithi116.stdout: 2023-12-03T11:58:23.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T11:58:23.195 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 STARTED 2023-12-03T11:58:23.225 INFO:tasks.rgw.client.0.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-03T11:58:28.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-12-03T11:58:34.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~6s 2023-12-03T11:58:40.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~13s 2023-12-03T11:58:47.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~19s 2023-12-03T11:58:53.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~25s 2023-12-03T11:58:59.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~32s 2023-12-03T11:59:06.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~38s 2023-12-03T11:59:12.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~44s 2023-12-03T11:59:18.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~50s 2023-12-03T11:59:25.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~57s 2023-12-03T11:59:31.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~63s 2023-12-03T11:59:37.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~69s 2023-12-03T11:59:44.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~76s 2023-12-03T11:59:50.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~82s 2023-12-03T11:59:56.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~88s 2023-12-03T12:00:02.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~95s 2023-12-03T12:00:09.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~101s 2023-12-03T12:00:15.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~107s 2023-12-03T12:00:21.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~114s 2023-12-03T12:00:28.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~120s 2023-12-03T12:00:34.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~126s 2023-12-03T12:00:40.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~132s 2023-12-03T12:00:47.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~139s 2023-12-03T12:00:53.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~145s 2023-12-03T12:00:59.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~151s 2023-12-03T12:01:06.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~158s 2023-12-03T12:01:12.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~164s 2023-12-03T12:01:18.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~170s 2023-12-03T12:01:24.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~177s 2023-12-03T12:01:31.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~183s 2023-12-03T12:01:37.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~189s 2023-12-03T12:01:43.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~196s 2023-12-03T12:01:50.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~202s 2023-12-03T12:01:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~208s 2023-12-03T12:02:02.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~214s 2023-12-03T12:02:09.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~221s 2023-12-03T12:02:15.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~227s 2023-12-03T12:02:21.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~233s 2023-12-03T12:02:28.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~240s 2023-12-03T12:02:34.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~246s 2023-12-03T12:02:40.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~252s 2023-12-03T12:02:46.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~259s 2023-12-03T12:02:53.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~265s 2023-12-03T12:02:59.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~271s 2023-12-03T12:03:05.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~278s 2023-12-03T12:03:12.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~284s 2023-12-03T12:03:18.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~290s 2023-12-03T12:03:24.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~296s 2023-12-03T12:03:31.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~303s 2023-12-03T12:03:31.102 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-03T12:03:32.408 INFO:tasks.ceph.osd.0:Sent signal 15 2023-12-03T12:03:32.409 INFO:tasks.ceph.osd.1:Sent signal 15 2023-12-03T12:03:32.409 INFO:tasks.ceph.osd.2:Sent signal 15 2023-12-03T12:03:32.410 INFO:tasks.ceph.osd.3:Sent signal 15 2023-12-03T12:03:32.410 INFO:tasks.ceph.osd.4:Sent signal 15 2023-12-03T12:03:32.410 INFO:tasks.ceph.osd.5:Sent signal 15 2023-12-03T12:03:32.410 INFO:tasks.ceph.osd.6:Sent signal 15 2023-12-03T12:03:32.411 INFO:tasks.ceph.osd.7:Sent signal 15 2023-12-03T12:03:32.411 INFO:tasks.ceph.mon.a:Sent signal 15 2023-12-03T12:03:32.411 INFO:tasks.ceph.mon.c:Sent signal 15 2023-12-03T12:03:32.411 INFO:tasks.ceph.mon.b:Sent signal 15 2023-12-03T12:03:32.411 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-12-03T12:03:32.412 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-12-03T12:03:32.413 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff2f0a76640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83892) UID: 0 2023-12-03T12:03:32.413 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff2f0a76640 -1 osd.4 255 *** Got signal Terminated *** 2023-12-03T12:03:32.413 INFO:tasks.ceph.osd.4.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff2f0a76640 -1 osd.4 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.413 INFO:tasks.ceph.mon.b.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f61f06e8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 83365) UID: 0 2023-12-03T12:03:32.414 INFO:tasks.ceph.mon.b.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f61f06e8640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-03T12:03:32.414 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T12:03:32.408+0000 7f27a10fa640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83832) UID: 0 2023-12-03T12:03:32.414 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T12:03:32.408+0000 7f27a10fa640 -1 osd.0 255 *** Got signal Terminated *** 2023-12-03T12:03:32.414 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-12-03T12:03:32.408+0000 7f27a10fa640 -1 osd.0 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f1915b9b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83895) UID: 0 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f1915b9b640 -1 osd.5 255 *** Got signal Terminated *** 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.5.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f1915b9b640 -1 osd.5 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T12:03:32.408+0000 7efea4de1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83839) UID: 0 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T12:03:32.408+0000 7efea4de1640 -1 osd.1 255 *** Got signal Terminated *** 2023-12-03T12:03:32.415 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-12-03T12:03:32.408+0000 7efea4de1640 -1 osd.1 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f2d6e9ae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 83897) UID: 0 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f2d6e9ae640 -1 osd.6 255 *** Got signal Terminated *** 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.6.smithi152.stderr:2023-12-03T12:03:32.411+0000 7f2d6e9ae640 -1 osd.6 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f2a1e4bc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83847) UID: 0 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f2a1e4bc640 -1 osd.2 255 *** Got signal Terminated *** 2023-12-03T12:03:32.416 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f2a1e4bc640 -1 osd.2 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff228ee2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 83898) UID: 0 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff228ee2640 -1 osd.7 255 *** Got signal Terminated *** 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.7.smithi152.stderr:2023-12-03T12:03:32.411+0000 7ff228ee2640 -1 osd.7 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.417 INFO:tasks.ceph.mgr.x.smithi152.stderr:2023-12-03T12:03:32.411+0000 7fa5e88cc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 83367) UID: 0 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f6ab0792640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83846) UID: 0 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f6ab0792640 -1 osd.3 255 *** Got signal Terminated *** 2023-12-03T12:03:32.417 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-12-03T12:03:32.409+0000 7f6ab0792640 -1 osd.3 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T12:03:32.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-12-03T12:03:32.410+0000 7f0fe44ec640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 83520) UID: 0 2023-12-03T12:03:32.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-12-03T12:03:32.410+0000 7f0fe44ec640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-03T12:03:32.418 INFO:tasks.ceph.mon.c.smithi116.stderr:2023-12-03T12:03:32.411+0000 7f2d0bf21640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 83522) UID: 0 2023-12-03T12:03:32.418 INFO:tasks.ceph.mon.c.smithi116.stderr:2023-12-03T12:03:32.411+0000 7f2d0bf21640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-03T12:03:32.613 INFO:tasks.ceph.mgr.y.smithi116.stderr:daemon-helper: command crashed with signal 15 2023-12-03T12:05:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:05:40.298 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 FAILED 2023-12-03T12:05:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:212 2023-12-03T12:05:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:212 2023-12-03T12:05:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:212 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 STARTED 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 FAILED 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:168 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:168 2023-12-03T12:06:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:168 2023-12-03T12:07:20.296 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:07:20.296 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 STARTED 2023-12-03T12:07:20.296 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:07:20.296 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 FAILED 2023-12-03T12:07:20.296 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:258 2023-12-03T12:07:20.297 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:258 2023-12-03T12:07:20.297 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:258 2023-12-03T12:08:10.295 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:08:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 STARTED 2023-12-03T12:08:10.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:08:10.395 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 FAILED 2023-12-03T12:08:10.395 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:398 2023-12-03T12:08:10.395 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:398 2023-12-03T12:08:10.395 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:398 2023-12-03T12:09:00.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:09:00.396 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 STARTED 2023-12-03T12:09:00.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:09:00.397 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 FAILED 2023-12-03T12:09:00.397 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:442 2023-12-03T12:09:00.397 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:442 2023-12-03T12:09:00.397 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:442 2023-12-03T12:09:50.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:09:50.398 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 STARTED 2023-12-03T12:09:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:09:50.398 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 FAILED 2023-12-03T12:09:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:304 2023-12-03T12:09:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:304 2023-12-03T12:09:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:304 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 STARTED 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 FAILED 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:586 2023-12-03T12:10:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:586 2023-12-03T12:10:40.398 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:586 2023-12-03T12:11:30.499 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:11:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED 2023-12-03T12:11:30.500 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:11:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 FAILED 2023-12-03T12:11:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: com.amazonaws.SdkClientException at AWS4Test.java:825 2023-12-03T12:11:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:825 2023-12-03T12:11:30.501 INFO:teuthology.orchestra.run.smithi116.stdout: Caused by: java.net.ConnectException at AWS4Test.java:825 2023-12-03T12:12:20.496 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:12:20.496 INFO:teuthology.orchestra.run.smithi116.stdout:---------------------------------------------------------------------- 2023-12-03T12:12:20.496 INFO:teuthology.orchestra.run.smithi116.stdout:| Results: FAILURE (26 tests, 18 successes, 8 failures, 0 skipped) | 2023-12-03T12:12:20.496 INFO:teuthology.orchestra.run.smithi116.stdout:---------------------------------------------------------------------- 2023-12-03T12:12:20.497 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.497 INFO:teuthology.orchestra.run.smithi116.stderr:26 tests completed, 8 failed 2023-12-03T12:12:20.581 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:12:20.581 INFO:teuthology.orchestra.run.smithi116.stdout:> Task :test FAILED 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr:FAILURE: Build failed with an exception. 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr:* What went wrong: 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr:Execution failed for task ':test'. 2023-12-03T12:12:20.583 INFO:teuthology.orchestra.run.smithi116.stderr:> There were failing tests. See the report at: file:///home/ubuntu/cephtest/s3-tests-java/build/reports/tests/test/index.html 2023-12-03T12:12:20.584 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.584 INFO:teuthology.orchestra.run.smithi116.stderr:* Try: 2023-12-03T12:12:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 2023-12-03T12:12:20.585 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:* Get more help at https://help.gradle.org 2023-12-03T12:12:20.586 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-12-03T12:12:20.586 INFO:teuthology.orchestra.run.smithi116.stdout:Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. 2023-12-03T12:12:20.586 INFO:teuthology.orchestra.run.smithi116.stdout:Use '--warning-mode all' to show the individual deprecation warnings. 2023-12-03T12:12:20.586 INFO:teuthology.orchestra.run.smithi116.stdout:See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings 2023-12-03T12:12:20.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:12:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:BUILD FAILED in 14m 52s 2023-12-03T12:12:20.587 INFO:teuthology.orchestra.run.smithi116.stdout:5 actionable tasks: 4 executed, 1 up-to-date 2023-12-03T12:12:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:12:20.899 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 94, in begin File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 340, in run_tests File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' 2023-12-03T12:12:20.973 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4d560ea2f9d646d098934cc36da9df82 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 94, in begin File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 340, in run_tests File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' 2023-12-03T12:12:20.975 DEBUG:teuthology.run_tasks:Unwinding manager s3tests-java 2023-12-03T12:12:20.986 DEBUG:tasks.s3tests_java:S3 Tests Java: END 2023-12-03T12:12:20.986 INFO:tasks.s3tests_java:S3 Tests Java: Cleaning up s3-tests-java... 2023-12-03T12:12:20.986 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/s3-tests-java 2023-12-03T12:12:21.052 INFO:tasks.s3tests_java:S3 Tests Java: Deleting S3 users... 2023-12-03T12:12:21.053 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph 2023-12-03T12:12:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T12:12:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T12:17:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:17:21.114 INFO:teuthology.orchestra.run.smithi116.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-03T12:17:21.115 ERROR:teuthology.run_tasks:Manager failed: s3tests-java Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:17:21.121 DEBUG:teuthology.run_tasks:Unwinding manager ragweed 2023-12-03T12:17:21.132 DEBUG:teuthology.run_tasks:Unwinding manager tox 2023-12-03T12:17:21.153 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/tox-venv 2023-12-03T12:17:21.217 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-12-03T12:17:21.227 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:17:21.229 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-12-03T12:17:21.229 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:17:21.229 INFO:tasks.rgw.client.0:Stopped 2023-12-03T12:17:21.230 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-12-03T12:17:21.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/vault-root-token 2023-12-03T12:17:21.343 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process', '--include-all'] 2023-12-03T12:17:21.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process', '--include-all'] 2023-12-03T12:17:21.343 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all 2023-12-03T12:17:21.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2023-12-03T12:17:21.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T12:22:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:22:21.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-12-03T12:22:21.438+0000 7f723ef20800 0 monclient(hunting): authenticate timed out after 300 2023-12-03T12:22:21.446 INFO:teuthology.orchestra.run.smithi116.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-03T12:22:21.446 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 226, in start_rgw File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all' 2023-12-03T12:22:21.473 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2023-12-03T12:22:21.495 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/ca/root.crt /home/ubuntu/cephtest/ca/root.key 2023-12-03T12:22:21.545 DEBUG:teuthology.orchestra.run.smithi116:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-12-03T12:22:21.616 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/ca/rgw.client.0.crt /home/ubuntu/cephtest/ca/rgw.client.0.key 2023-12-03T12:22:21.687 DEBUG:teuthology.orchestra.run.smithi116:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-12-03T12:22:21.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm /usr/share/pki/ca-trust-source/anchors/root.crt 2023-12-03T12:22:21.842 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-ca-trust 2023-12-03T12:22:22.619 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-03T12:22:22.633 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-03T12:22:22.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T12:24:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-03T12:24:22.683 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-03T12:24:22.704 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-03T12:24:22.704 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-03T12:24:22.704 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-03T12:24:22.704 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.711 INFO:tasks.ceph.osd.0:Stopped 2023-12-03T12:24:22.711 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-03T12:24:22.711 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.713 INFO:tasks.ceph.osd.1:Stopped 2023-12-03T12:24:22.713 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-03T12:24:22.713 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.713 INFO:tasks.ceph.osd.2:Stopped 2023-12-03T12:24:22.713 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-03T12:24:22.714 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.714 INFO:tasks.ceph.osd.3:Stopped 2023-12-03T12:24:22.714 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-03T12:24:22.714 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.714 INFO:tasks.ceph.osd.4:Stopped 2023-12-03T12:24:22.714 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-03T12:24:22.714 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.715 INFO:tasks.ceph.osd.5:Stopped 2023-12-03T12:24:22.715 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-03T12:24:22.715 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.715 INFO:tasks.ceph.osd.6:Stopped 2023-12-03T12:24:22.715 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-03T12:24:22.715 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.716 INFO:tasks.ceph.osd.7:Stopped 2023-12-03T12:24:22.716 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-03T12:24:22.717 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-03T12:24:22.717 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:24:22.718 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1447, in run_daemon File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-12-03T12:24:22.736 INFO:tasks.ceph.mgr.y:Stopped 2023-12-03T12:24:22.736 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-03T12:24:22.737 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.737 INFO:tasks.ceph.mgr.x:Stopped 2023-12-03T12:24:22.737 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-03T12:24:22.738 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-03T12:24:22.738 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.738 INFO:tasks.ceph.mon.a:Stopped 2023-12-03T12:24:22.739 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-03T12:24:22.739 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.739 INFO:tasks.ceph.mon.c:Stopped 2023-12-03T12:24:22.739 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-03T12:24:22.739 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:24:22.739 INFO:tasks.ceph.mon.b:Stopped 2023-12-03T12:24:22.740 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-03T12:24:22.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-03T12:24:22.870 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T12:24:22.871 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-03T12:24:23.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T12:24:23.008 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-03T12:24:23.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T12:24:23.096 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-03T12:24:23.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T12:24:23.176 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-03T12:24:23.258 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T12:24:23.258 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-03T12:24:23.321 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T12:24:23.321 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-03T12:24:23.378 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T12:24:23.378 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-03T12:24:23.460 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T12:24:23.460 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-03T12:24:23.517 INFO:tasks.ceph:Archiving mon data... 2023-12-03T12:24:23.519 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/data/mon.a.tgz 2023-12-03T12:24:23.519 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-12-03T12:24:23.547 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.Gy8jdljSwP 2023-12-03T12:24:23.548 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Gy8jdljSwP 2023-12-03T12:24:23.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.Gy8jdljSwP 2023-12-03T12:24:23.828 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.Gy8jdljSwP is 343KB 2023-12-03T12:24:23.929 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.Gy8jdljSwP 2023-12-03T12:24:23.956 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/data/mon.c.tgz 2023-12-03T12:24:23.957 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-12-03T12:24:24.024 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.nZNE4NleG4 2023-12-03T12:24:24.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.nZNE4NleG4 2023-12-03T12:24:24.188 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.nZNE4NleG4 2023-12-03T12:24:24.298 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.nZNE4NleG4 is 345KB 2023-12-03T12:24:24.395 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.nZNE4NleG4 2023-12-03T12:24:24.422 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/data/mon.b.tgz 2023-12-03T12:24:24.423 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2023-12-03T12:24:24.441 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.iBT9mh4iOE 2023-12-03T12:24:24.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.iBT9mh4iOE 2023-12-03T12:24:24.616 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.iBT9mh4iOE 2023-12-03T12:24:24.703 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.iBT9mh4iOE is 345KB 2023-12-03T12:24:24.801 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.iBT9mh4iOE 2023-12-03T12:24:24.816 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-03T12:24:24.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-03T12:24:24.819 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-03T12:24:24.924 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-03T12:24:24.939 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-03T12:24:24.939 INFO:tasks.ceph:Archiving crash dumps... 2023-12-03T12:24:24.940 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi116/crash 2023-12-03T12:24:24.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-03T12:24:25.062 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi152/crash 2023-12-03T12:24:25.063 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-03T12:24:25.089 INFO:tasks.ceph:Compressing logs... 2023-12-03T12:24:25.089 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-03T12:24:25.092 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-03T12:24:25.172 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-03T12:24:25.173 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83562.log 2023-12-03T12:24:25.173 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83682.log 2023-12-03T12:24:25.173 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83562.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-03T12:24:25.174 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83562.log.gz 2023-12-03T12:24:25.174 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83722.log 2023-12-03T12:24:25.174 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83682.log.gz 2023-12-03T12:24:25.174 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83602.log 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83722.log.gz 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83602.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83602.log.gz 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83522.log 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-03T12:24:25.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83642.log 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83522.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.83802.log -- replaced with /var/log/ceph/ceph-client.admin.83522.log.gz 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr: 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83642.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83642.log.gz 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83762.log 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83802.log.gz 2023-12-03T12:24:25.176 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-03T12:24:25.177 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.83762.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83762.log.gz 2023-12-03T12:24:25.177 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-03T12:24:25.177 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-03T12:24:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93614.log 2023-12-03T12:24:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98931.log 2023-12-03T12:24:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110213.log 2023-12-03T12:24:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93614.log: /var/log/ceph/ceph-client.admin.98931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104431.log 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.93614.log.gz 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr: 80.8% -- replaced with /var/log/ceph/ceph-client.admin.98931.log.gz 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.110213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101561.log 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110213.log.gz 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104431.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.104431.log.gz 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94318.log 2023-12-03T12:24:25.278 INFO:teuthology.orchestra.run.smithi116.stderr: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.101561.log.gz 2023-12-03T12:24:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.95847.log 2023-12-03T12:24:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94236.log 2023-12-03T12:24:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95847.log: gzip -5 84.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.94318.log.gz 2023-12-03T12:24:25.279 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.93908.log 2023-12-03T12:24:25.279 INFO:teuthology.orchestra.run.smithi116.stderr: 43.2% -- replaced with /var/log/ceph/ceph-client.admin.95847.log.gz 2023-12-03T12:24:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102135.log 2023-12-03T12:24:25.280 INFO:teuthology.orchestra.run.smithi116.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.94236.log.gz 2023-12-03T12:24:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104924.log 2023-12-03T12:24:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91286.log 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.93908.log.gz 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.102135.log.gz 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90872.log 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91286.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.104924.log.gz /var/log/ceph/ceph-client.admin.89912.log 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90872.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.90872.log.gz --verbose 2023-12-03T12:24:25.282 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.89920.log 2023-12-03T12:24:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105708.log 2023-12-03T12:24:25.282 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89912.log.gz 2023-12-03T12:24:25.282 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.0.91286.log.gz 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89920.log.gz 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103119.log 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93990.log 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105708.log.gz 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89916.log 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.103119.log.gz 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98848.log 2023-12-03T12:24:25.283 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.93990.log.gz 2023-12-03T12:24:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99582.log 2023-12-03T12:24:25.284 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89916.log.gz 2023-12-03T12:24:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94646.log 2023-12-03T12:24:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.99582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91674.log 2023-12-03T12:24:25.285 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98848.log.gz 2023-12-03T12:24:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94646.log.gz 2023-12-03T12:24:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93826.log 2023-12-03T12:24:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89513.log 2023-12-03T12:24:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91674.log: /var/log/ceph/ceph-client.admin.93826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83562.log 2023-12-03T12:24:25.290 INFO:teuthology.orchestra.run.smithi116.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.0.91674.log.gz 2023-12-03T12:24:25.290 INFO:teuthology.orchestra.run.smithi116.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.93826.log.gz 2023-12-03T12:24:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89513.log.gz 2023-12-03T12:24:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108890.log 2023-12-03T12:24:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96420.log 2023-12-03T12:24:25.293 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83562.log.gz 2023-12-03T12:24:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97518.log 2023-12-03T12:24:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.smithi116.109427.log 2023-12-03T12:24:25.293 INFO:teuthology.orchestra.run.smithi116.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.96420.log.gz 2023-12-03T12:24:25.293 INFO:teuthology.orchestra.run.smithi116.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.99582.log.gz 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102791.log 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98069.log 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.crash.smithi116.109427.log: 52.5% -- replaced with /var/log/ceph/ceph-client.crash.smithi116.109427.log.gz 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106026.log 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98069.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102791.log.gz 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91181.log 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106026.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108890.log.gz 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102299.log 2023-12-03T12:24:25.294 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106026.log.gz 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.91181.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.102381.log 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91181.log.gz 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101233.log 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103611.log 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.102299.log.gz 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.102381.log.gz 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103529.log 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.98069.log.gz 2023-12-03T12:24:25.295 INFO:teuthology.orchestra.run.smithi116.stderr: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.101233.log.gz 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91099.log 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.103611.log.gz 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90452.log 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.91099.log: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.103529.log.gz 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91099.log.gz 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101643.log 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94670.log 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90452.log.gz 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97629.log 2023-12-03T12:24:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101315.log 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94670.log.gz 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95070.log 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105274.log 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.101315.log.gz 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.97518.log.gz 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.101643.log.gz 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91263.log 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100741.log 2023-12-03T12:24:25.297 INFO:teuthology.orchestra.run.smithi116.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.95070.log.gz 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105274.log.gz 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.91263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89738.log 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91263.log.gz 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90788.log 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100741.log: /var/log/ceph/ceph-client.admin.89738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98761.log 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89738.log.gz 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100741.log.gz 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90788.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.0.91484.log 2023-12-03T12:24:25.298 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90788.log.gz 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100331.log 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.98761.log.gz 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105978.log 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.0.91484.log.gz 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90599.log 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100331.log.gz 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90512.log 2023-12-03T12:24:25.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105978.log.gz 2023-12-03T12:24:25.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93402.log 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90599.log.gz 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90512.log.gz 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90253.log 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.97629.log.gz 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91221.log 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93402.log.gz 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99342.log 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90253.log.gz 2023-12-03T12:24:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.91221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95046.log 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91221.log.gz 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.99342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.109439.log 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95046.log: 84.0% -- replaced with /var/log/ceph/ceph-client.0.99342.log.gz 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105602.log 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95046.log.gz 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.crash.109439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93778.log 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr: 13.0% -- replaced with /var/log/ceph/ceph-client.crash.109439.log.gz 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105088.log 2023-12-03T12:24:25.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105602.log.gz 2023-12-03T12:24:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103775.log 2023-12-03T12:24:25.303 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93778.log.gz 2023-12-03T12:24:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100495.log 2023-12-03T12:24:25.303 INFO:teuthology.orchestra.run.smithi116.stderr: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.105088.log.gz 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103775.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.103775.log.gz 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93238.log 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.100495.log.gz 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104678.log 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96996.log 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.93238.log.gz 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105732.log 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96996.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104678.log.gz 2023-12-03T12:24:25.304 INFO:teuthology.orchestra.run.smithi116.stderr: 19.9% -- replaced with /var/log/ceph/ceph-client.admin.96996.log.gz 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90598.log 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104842.log 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105732.log: /var/log/ceph/ceph-client.admin.90598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98289.log 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90598.log.gz 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105732.log.gz 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94482.log 2023-12-03T12:24:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104842.log: /var/log/ceph/ceph-client.admin.98289.log: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.104842.log.gz 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101069.log 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr: 94.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108991.log 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94482.log: /var/log/ceph/ceph-client.admin.101069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83853.log 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.94482.log.gz 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.101069.log.gz 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98289.log.gz 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99244.log 2023-12-03T12:24:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102955.log 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.108991.log.gz 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83853.log.gz 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.99244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.108748.log 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.0.99244.log.gz 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101151.log 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.102955.log.gz 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.108748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97959.log 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr: 83.9% -- replaced with /var/log/ceph/ceph-client.0.108748.log.gz 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91392.log 2023-12-03T12:24:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90173.log 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101151.log.gz 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109924.log 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr: 43.5% -- replaced with /var/log/ceph/ceph-client.0.91392.log.gz 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104595.log 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90173.log.gz 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.109924.log: 9.2% -- replaced with /var/log/ceph/ceph-client.0.109924.log.gz 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99015.log 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90256.log 2023-12-03T12:24:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104595.log: /var/log/ceph/ceph-client.admin.99015.log: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.104595.log.gz 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97849.log 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.99015.log.gz 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91864.log 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90256.log.gz 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.97849.log: --verbose -- /var/log/ceph/ceph-client.admin.104021.log 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97739.log 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr: 43.5% -- replaced with /var/log/ceph/ceph-client.0.91864.log.gz 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100987.log 2023-12-03T12:24:25.309 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9%/var/log/ceph/ceph-client.admin.97739.log: -- replaced with /var/log/ceph/ceph-client.admin.104021.log.gz 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100577.log 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100987.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97849.log.gz 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105006.log 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr: 81.2% -- replaced with /var/log/ceph/ceph-client.admin.100987.log.gz 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89915.log 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.100577.log.gz 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94154.log 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89915.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97739.log.gz 2023-12-03T12:24:25.310 INFO:teuthology.orchestra.run.smithi116.stderr: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.105006.log.gz 2023-12-03T12:24:25.311 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89915.log.gz 2023-12-03T12:24:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100823.log 2023-12-03T12:24:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94154.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-03T12:24:25.311 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.94154.log.gz 2023-12-03T12:24:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94400.log 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.100823.log.gz 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101397.log 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90252.log 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.94400.log.gz 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102053.log 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103857.log 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90252.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101397.log.gz 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90252.log.gz 2023-12-03T12:24:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97298.log 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr: 81.5% -- replaced with /var/log/ceph/ceph-client.admin.102053.log.gz 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100167.log 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103857.log.gz 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102545.log 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104760.log 2023-12-03T12:24:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100167.log: 78.1% -- replaced with /var/log/ceph/ceph-client.admin.97298.log.gz 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110016.log 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr: 81.3% -- replaced with /var/log/ceph/ceph-client.admin.100167.log.gz 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104513.log 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104760.log: /var/log/ceph/ceph-client.0.110016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101807.log 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.104760.log.gz 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr: 11.2% -- replaced with /var/log/ceph/ceph-client.0.110016.log.gz 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.77699.log 2023-12-03T12:24:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104513.log: /var/log/ceph/ceph-client.admin.101807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96305.log 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.102545.log.gz 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr: 81.3% -- replaced with /var/log/ceph/ceph-client.admin.101807.log.gz 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104103.log 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.77699.log: /var/log/ceph/ceph-client.admin.96305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89424.log 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.77699.log.gz 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95957.log 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89424.log: 84.9% 86.2% -- replaced with /var/log/ceph/ceph-client.admin.104513.log.gz 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104103.log.gz 2023-12-03T12:24:25.315 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94694.log 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89424.log.gz 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90510.log 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95957.log: /var/log/ceph/ceph-client.admin.94694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102627.log 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr: 58.6% -- replaced with /var/log/ceph/ceph-client.admin.95957.log.gz 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr: 85.3%/var/log/ceph/ceph-client.admin.90510.log: -- replaced with /var/log/ceph/ceph-client.admin.94694.log.gz 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89570.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90510.log.gz 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102627.log: gzip -5 --verbose -- 81.8% -- replaced with /var/log/ceph/ceph-client.admin.102627.log.gz 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.90166.log 2023-12-03T12:24:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89570.log.gz 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102217.log 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90166.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90166.log.gz 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.93320.log 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102217.log: gzip -5 --verbose -- 81.8% -- replaced with /var/log/ceph/ceph-client.admin.102217.log.gz 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.100003.log 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100085.log 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.93320.log.gz 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93074.log 2023-12-03T12:24:25.317 INFO:teuthology.orchestra.run.smithi116.stderr: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.100003.log.gz 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100085.log: gzip -5 --verbose -- 81.6% -- replaced with /var/log/ceph/ceph-client.admin.100085.log.gz 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.105253.log 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105684.log 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.93074.log.gz 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105253.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105253.log.gz 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.0.91582.log 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105684.log.gzgzip 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.93802.log 2023-12-03T12:24:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100659.log 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91582.log.gz 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93802.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93802.log.gz 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96175.log 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.100659.log.gz 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103939.log 2023-12-03T12:24:25.319 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.96175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89918.log 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0%/var/log/ceph/ceph-client.admin.103939.log: -- replaced with /var/log/ceph/ceph-client.admin.96175.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104267.log 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.103939.log.gz 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99680.log 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89918.log.gz 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100905.log 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.0.99680.log: /var/log/ceph/ceph-client.admin.95581.log 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104267.log.gz 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100905.log: 80.8% -- replaced with /var/log/ceph/ceph-client.0.99680.log.gz 2023-12-03T12:24:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93426.log 2023-12-03T12:24:25.321 INFO:teuthology.orchestra.run.smithi116.stderr: 81.0% -- replaced with /var/log/ceph/ceph-client.admin.100905.log.gz 2023-12-03T12:24:25.321 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97408.log 2023-12-03T12:24:25.321 INFO:teuthology.orchestra.run.smithi116.stderr: 94.8% -- replaced with /var/log/ceph/ceph-client.admin.97959.log.gz 2023-12-03T12:24:25.321 INFO:teuthology.orchestra.run.smithi116.stderr: 60.5% -- replaced with /var/log/ceph/ceph-client.admin.95581.log.gz 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95234.log 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93426.log.gz 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91772.log 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89919.log 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.95234.log.gz 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91772.log: 42.8% -- replaced with /var/log/ceph/ceph-client.0.91772.log.gz 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.admin.log 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90171.log 2023-12-03T12:24:25.322 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89919.log: 0.0%/var/log/ceph/ops-log-ceph-client.admin.log: -- replaced with /var/log/ceph/ceph-client.admin.89919.log.gz 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103693.log 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90171.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105814.log 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103693.log: /var/log/ceph/ceph-mgr.y.log: 87.9% 86.9% -- replaced with /var/log/ceph/ceph-client.admin.103693.log.gz 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96305.log.gz 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99530.log 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83974.log 2023-12-03T12:24:25.323 INFO:teuthology.orchestra.run.smithi116.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105814.log.gz 2023-12-03T12:24:25.324 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99530.log: 50.1% -- replaced with /var/log/ceph/ceph-client.admin.99530.log.gz 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 94.2% -- replaced with /var/log/ceph/ops-log-ceph-client.admin.log.gz 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.103201.log 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83974.log.gz 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98179.log 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.97408.log.gz 2023-12-03T12:24:25.325 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102463.log 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103201.log: /var/log/ceph/ceph-client.admin.98179.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103201.log.gz 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.101971.log 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96364.log 81.8% -- replaced with /var/log/ceph/ceph-client.admin.102463.log.gz 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101971.log: gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.98179.log.gz 2023-12-03T12:24:25.326 INFO:teuthology.orchestra.run.smithi116.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.101971.log.gz 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90589.log 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103037.log 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr: 55.2% -- replaced with /var/log/ceph/ceph-client.admin.96364.log.gz 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90589.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90589.log.gz 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90511.log 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101725.log 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90511.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.103037.log.gz 2023-12-03T12:24:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90511.log.gz 2023-12-03T12:24:25.328 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102873.log 2023-12-03T12:24:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90594.log 2023-12-03T12:24:25.328 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.101725.log.gz 2023-12-03T12:24:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98679.log 2023-12-03T12:24:25.329 INFO:teuthology.orchestra.run.smithi116.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.102873.log.gz 2023-12-03T12:24:25.329 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90594.log.gz 2023-12-03T12:24:25.329 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100413.log 2023-12-03T12:24:25.329 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100249.log 2023-12-03T12:24:25.329 INFO:teuthology.orchestra.run.smithi116.stderr: 80.6% -- replaced with /var/log/ceph/ceph-client.admin.98679.log.gz 2023-12-03T12:24:25.330 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91139.log 2023-12-03T12:24:25.330 INFO:teuthology.orchestra.run.smithi116.stderr: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.100413.log.gz 2023-12-03T12:24:25.330 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109451.log 2023-12-03T12:24:25.331 INFO:teuthology.orchestra.run.smithi116.stderr: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.100249.log.gz 2023-12-03T12:24:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.91139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91139.log.gz 2023-12-03T12:24:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90254.log 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109451.log.gz 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94776.log 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90254.log.gz 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105170.log 2023-12-03T12:24:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94564.log 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.94776.log.gz 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106002.log 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr: 80.7% -- replaced with /var/log/ceph/ceph-client.admin.105170.log.gz 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.94564.log: --verbose -- /var/log/ceph/ops-log-ceph-client.0.log 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.94564.log.gz 2023-12-03T12:24:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105438.log 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106002.log: /var/log/ceph/ops-log-ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105896.log 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106002.log.gz 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93450.log 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.105438.log.gz 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102709.log 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93450.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105896.log.gz 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.93450.log.gz 2023-12-03T12:24:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106108.log 2023-12-03T12:24:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89917.log 2023-12-03T12:24:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106108.log: /var/log/ceph/ceph-client.admin.102709.log: 81.0% -- replaced with /var/log/ceph/ceph-client.admin.102709.log.gz 2023-12-03T12:24:25.335 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106108.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101889.log 2023-12-03T12:24:25.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-03T12:24:25.339 INFO:teuthology.orchestra.run.smithi116.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-03T12:24:25.339 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89917.log: /var/log/ceph/ceph-client.admin.101889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89917.log.gz 2023-12-03T12:24:25.339 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101889.log.gz 2023-12-03T12:24:25.339 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.106190.log 2023-12-03T12:24:25.339 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104185.log 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89654.log 82.1% -- replaced with /var/log/ceph/ceph-client.admin.106190.log.gz 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.92048.log 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89654.log: /var/log/ceph/ceph-client.admin.104185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89654.log.gz 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.104185.log.gz 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.92048.log: 43.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103283.log 2023-12-03T12:24:25.340 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.0.92048.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97188.log 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103283.log: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.103283.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89908.log 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr: 93.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103447.log 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89908.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93696.log 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103447.log: 94.5% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103447.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ops-log-ceph-client.0.log.gz 2023-12-03T12:24:25.341 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101479.log 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95152.log 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.93696.log.gz 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101479.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95152.log: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.101479.log.gz 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94940.log 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.95152.log.gz 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90830.log 2023-12-03T12:24:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94940.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.94940.log.gz 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93532.log 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90830.log.gz 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104349.log 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93532.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.93532.log.gz 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91956.log 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104349.log: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.104349.log.gz 2023-12-03T12:24:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93156.log 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.91956.log: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91956.log.gz 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105520.log 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93156.log: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.93156.log.gz 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89612.log 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105520.log: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.105520.log.gz 2023-12-03T12:24:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96249.log 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89612.log.gz 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94858.log 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96249.log.gz 93.8% -- replaced with /var/log/ceph/ceph-client.admin.97188.log.gz 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105356.log 2023-12-03T12:24:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103365.log 2023-12-03T12:24:25.346 INFO:teuthology.orchestra.run.smithi116.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.94858.log.gz 2023-12-03T12:24:25.346 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97106.log 2023-12-03T12:24:25.346 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103365.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.105356.log.gz 2023-12-03T12:24:25.346 INFO:teuthology.orchestra.run.smithi116.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103365.log.gz 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94072.log 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89696.log 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97106.log: /var/log/ceph/ceph-client.admin.94072.log: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.97106.log.gz 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.94072.log.gz 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95022.log 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99099.log 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89696.log.gz 2023-12-03T12:24:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95022.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90176.log 2023-12-03T12:24:25.348 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-03T12:24:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.90176.log: /var/log/ceph/ceph-client.admin.99099.log: 0.0% 25.3% -- replaced with /var/log/ceph/ceph-client.admin.99099.log.gz 2023-12-03T12:24:25.348 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90176.log.gz 2023-12-03T12:24:25.390 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-03T12:24:25.482 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2023-12-03T12:24:25.660 INFO:teuthology.orchestra.run.smithi116.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-03T12:24:25.692 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-03T12:24:26.408 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-03T12:24:28.005 INFO:teuthology.orchestra.run.smithi152.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-03T12:24:28.150 INFO:teuthology.orchestra.run.smithi116.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-03T12:24:28.322 INFO:teuthology.orchestra.run.smithi152.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-03T12:24:28.780 INFO:teuthology.orchestra.run.smithi116.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-03T12:24:29.490 INFO:teuthology.orchestra.run.smithi152.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-03T12:24:30.426 INFO:teuthology.orchestra.run.smithi116.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-03T12:24:32.222 INFO:teuthology.orchestra.run.smithi152.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-03T12:24:32.224 INFO:teuthology.orchestra.run.smithi152.stderr: 2023-12-03T12:24:32.224 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m7.076s 2023-12-03T12:24:32.225 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m17.508s 2023-12-03T12:24:32.225 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.646s 2023-12-03T12:24:36.921 INFO:teuthology.orchestra.run.smithi116.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-03T12:24:36.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-12-03T12:24:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m11.807s 2023-12-03T12:24:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m24.874s 2023-12-03T12:24:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.977s 2023-12-03T12:24:36.924 INFO:tasks.ceph:Archiving logs... 2023-12-03T12:24:36.924 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi116/log 2023-12-03T12:24:36.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-03T12:24:40.761 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi152/log 2023-12-03T12:24:40.762 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-03T12:24:43.546 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-03T12:24:43.560 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-03T12:24:43.578 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:24:43.600 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-03T12:24:43.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-03T12:24:43.606 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-03T12:24:43.703 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-03T12:24:43.714 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-03T12:24:43.715 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T12:24:43.717 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T12:24:43.734 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2023-12-03T12:24:43.739 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T12:24:43.739 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2023-12-03T12:24:43.739 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +2579us[+2579us] +/- 85ms 2023-12-03T12:24:43.739 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -977us[ -961us] +/- 66ms 2023-12-03T12:24:43.739 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -1102us[-1085us] +/- 73ms 2023-12-03T12:24:43.740 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T12:24:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-12-03T12:24:43.748 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T12:24:43.748 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-12-03T12:24:43.748 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +2571us[+2571us] +/- 84ms 2023-12-03T12:24:43.748 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -993us[ -993us] +/- 66ms 2023-12-03T12:24:43.749 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -1122us[-1122us] +/- 73ms 2023-12-03T12:24:43.749 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T12:24:43.750 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-03T12:24:43.761 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-03T12:24:43.762 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-03T12:24:43.773 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-03T12:24:43.900 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-03T12:24:43.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T12:24:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:24:44.044 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T12:24:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:24:44.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-03T12:24:44.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-03T12:24:44.102 INFO:teuthology.task.internal:Duration was 3004.427389 seconds 2023-12-03T12:24:44.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-03T12:24:44.117 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-03T12:24:44.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-03T12:24:44.121 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-03T12:24:44.167 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T12:24:44.174 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T12:24:44.348 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-03T12:24:44.349 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-12-03T12:24:44.349 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-03T12:24:44.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2023-12-03T12:24:44.381 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-03T12:24:44.404 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-03T12:24:44.404 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-03T12:24:44.425 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-03T12:24:44.473 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-03T12:24:44.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-03T12:24:44.505 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-03T12:24:44.612 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-03T12:24:44.627 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-03T12:24:44.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-03T12:24:44.667 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-03T12:24:44.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-03T12:24:44.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:24:44.710 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:24:44.753 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-12-03T12:24:44.763 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2023-12-03T12:24:44.801 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:24:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:24:44.832 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:24:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:24:44.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-03T12:24:44.870 INFO:teuthology.task.internal:Transferring archived files... 2023-12-03T12:24:44.871 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi116 2023-12-03T12:24:44.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-03T12:24:44.927 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150/remote/smithi152 2023-12-03T12:24:44.928 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-03T12:24:44.967 INFO:teuthology.task.internal:Removing archive directory... 2023-12-03T12:24:44.967 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-03T12:24:44.970 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-03T12:24:45.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-03T12:24:45.029 INFO:teuthology.task.internal:Not uploading archives. 2023-12-03T12:24:45.029 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-03T12:24:45.052 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-03T12:24:45.052 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-03T12:24:45.055 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-03T12:24:45.071 INFO:teuthology.orchestra.run.smithi152.stdout: 265321 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 12:24 /home/ubuntu/cephtest 2023-12-03T12:24:45.080 INFO:teuthology.orchestra.run.smithi116.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 12:24 /home/ubuntu/cephtest 2023-12-03T12:24:45.081 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-03T12:24:45.097 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-03T12:24:45.116 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi116 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:24:45.116 INFO:teuthology.nuke:Checking targets against current locks 2023-12-03T12:24:45.145 DEBUG:teuthology.nuke:shortname: smithi116 2023-12-03T12:24:45.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T12:24:45.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:26:17.977366', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWklY4I2diZT214tI3xTz6GRNhwlcyPni6UlOvuW/o+2IoxhJOzik5ptxGmaxuz2N7CpKMa/Odz5B5mVb3znSg='} 2023-12-03T12:24:45.176 DEBUG:teuthology.nuke:shortname: smithi152 2023-12-03T12:24:45.177 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T12:24:45.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:26:17.975308', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsJ+OYRQMdJGRs6a9/kmeo3WSNgQsUqBVDEgU4jFR4A5K8Xqv6bW0WMEOXZjF3goVFqedsSiXKs+J2O59oiA7w='} 2023-12-03T12:24:45.235 INFO:teuthology.orchestra.console.smithi116:Power off 2023-12-03T12:24:45.236 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T12:24:45.308 INFO:teuthology.orchestra.console.smithi152:Power off 2023-12-03T12:24:45.309 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T12:24:45.328 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2023-12-03T12:24:45.328 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:45.347 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2023-12-03T12:24:45.347 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:45.372 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-12-03T12:24:45.390 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-12-03T12:24:49.373 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:49.391 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:49.486 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-12-03T12:24:49.507 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-12-03T12:24:53.497 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:53.510 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:24:53.612 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2023-12-03T12:24:53.627 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-12-03T12:24:53.715 INFO:teuthology.orchestra.console.smithi116:Power off completed 2023-12-03T12:24:53.728 INFO:teuthology.orchestra.console.smithi152:Power off completed 2023-12-03T12:24:53.833 INFO:teuthology.run:Summary data: description: rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} duration: 3004.4273891448975 failure_reason: 'Command failed on smithi116 with status 1: ''cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test''' flavor: default owner: scheduled_alimasa@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4d560ea2f9d646d098934cc36da9df82 status: fail success: false 2023-12-03T12:24:53.833 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T12:24:53.997 INFO:teuthology.run:FAIL